Ungate the VNNI path in mul_mat_q8_1_r8_q8_2 by changing the
guard from HAVE_FANCY_SIMD to HAVE_VNNI256. This block only uses
256-bit intrinsics so it is safe for AVX-VNNI (non-512) CPUs.
Co-authored-by: Adam Caldwell <accaldwell@users.noreply.github.com>
* Allow using -rtr and -muge together
* Various Qwen-3.5 tweaks
* No need to make v, g, beta contiguous
* Adjust NEON delta-net to non-contiguous v, g, b
* Cleanup
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Without this, libcpp-httplib.a is compiled without SSL support, causing
an undefined reference to httplib::SSLServer at link time even though
the OpenSSL libraries are present on the link line.
Fixes#1449
Co-authored-by: kerem seyhan <kerem.seyhan@codecut.de>
Add new CPU macro HAVE_VNNI256 for CPUs with 256-bit VNNI
(AVX-VNNI) support or better (AVX512-VNNI+VL), separate from
HAVE_FANCY_SIMD which requires the full AVX-512 set. Relax four
#ifdef guards in mul_mat_q4_k_r4_q8_k and mul_mat_q5_k_r4_q8_k
to use HAVE_VNNI256 instead of HAVE_FANCY_SIMD, enabling vpdpbusd
and cvtepi8_epi32 on Alder Lake, Raptor Lake, and similar CPUs.
Co-authored-by: Adam Caldwell <accaldwell@users.noreply.github.com>
* Improve text formatting
* Update README.md with recent models and features
* Update parameters.md with recent additions
* Remove deprecated from parameters.md
* simpler n_rewind logic, delete old comments
* use more consistent names, add updt_w_cur to json schema
* align comments
* refactor review logic, update struct/variable names
* revert cosmetic changes
* check enable/disable in llama_prep_adaptive_p_impl()
* delete extra whitespaces after statement
* show target in debug prints
* more concise debug print
* delete old comments
* update with loop instead of move()
* comment out all adaptive p debug prints
* more debug prints
* move review() variables: common_sampler struct -> common_sampler_review() args
* match n_unsent type
* fix merge bugs, delete adaptive p references in buffer_and_check_string_ban()
* restore accidental erasure
* Revert "adaptive p: collect probability before logit bias"
This reverts commit 1434878461c49d1a2a9047fc15d5e7b78421fd2a.
* Use cuda 86 instead of default
"default" fails to build
* Update docker README.md
- Use 86 architecture
- Examples for mix of architectures
- Where to identify Cuda version
- Hint to clean unused images
- How to build without llama-swap
* Improve compatibility with the new Jinja template engine
* Refactor MiroThinker chat template using macros
* Add MiroThinker-compat chat template
For compatibility reasons. It can be removed when upstream fix the jinja render engine bugs.
* Add workaround for llama.cpp crashing
* server : support multi-modal context checkpoints and prompt caching
do not create checkpoint right after image processing
improve mtmd check for slot ops
fix context shift
do not abort if template parse failed
* change to debug message when detecting ban token
---------
Co-authored-by: firecoperana <firecoperana>
* WIP: support pre-merged up/gate experts
Haha, mainline has elected to arrange the merged tensors
the other way around compared to what I had done in the on-the-fly merge.
* Change the order of on-the-fly packed up/gate
* OpenAI
* CUDA TG
* CPU
* Split mode graph for models with pre-merged ffn_up/ffn_gate experts
* WIP: support pre-merged up/gate experts
Haha, mainline has elected to arrange the merged tensors
the other way around compared to what I had done in the on-the-fly merge.
* Change the order of on-the-fly packed up/gate
* OpenAI
* CUDA TG
* CPU
Two changes:
1. Add four missing environment variable bindings to
gpt_params_parse_from_env():
- LLAMA_ARG_CACHE_TYPE_K (string, e.g. "q8_0")
- LLAMA_ARG_CACHE_TYPE_V (string, e.g. "q8_0")
- LLAMA_ARG_MLOCK (bool, "1"/"true")
- LLAMA_ARG_K_CACHE_HADAMARD (bool, "1"/"true")
2. Call gpt_params_parse_from_env() from gpt_params_parse() so that
ALL tools (llama-cli, llama-bench, etc.) respect env vars, not
just llama-server. Env vars act as defaults; CLI flags override.
Follows the existing get_env() pattern and uses the same
LLAMA_ARG_ prefix convention as the other env vars.
Co-authored-by: Pipboyguy <>
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
Loads and starts running, crashes with illegal memory access in
quantize_mmq_q8_1. This almost always indicates NaNs in the input
to the MoE FFN part.
* WIP
* WIP
Loads and runs, wrong results (very high PPL)
Performance looks promising, around 25% better than previous sm graph.
Needs f32 or bf16 graph reduce type.
* WIP - still wrong
* Fix after rebase
* WIP
* WIP
* This seems to be working for dense Qwen3.5!!!
* WIP: Qwen3-Next is not quite working
* Some cleanup
* Disable Qwen3-Next for now
* Disable graph parallel when mmproj was specified
* Read/write split recurrent state
* That should not crash
* Re-enable vision - it works now
* Recurrent layers should now be counted for split cache
* Fix compilation on clang-cl.exe
Fixes https://github.com/ikawrakow/ik_llama.cpp/issues/1169
See bitwise ariphmetics here: https://clang.llvm.org/doxygen/avx512fintrin_8h_source.html
Clang (and GCC) supports a language feature called Vector Extensions.
To Clang, `__m512i` is not just a "struct" or a "bag of bits"; it is recognized by the compiler as a native vector type.
Because it is a native vector type, Clang automatically maps standard C operators to the corresponding hardware instructions.
When you write `a | b`, Clang sees that a and b are 512-bit integer vectors.
It implicitly understands that the bitwise OR operator (|) applies to these vectors.
It automatically generates the VPORQ (or VPORD) instruction without needing any helper function.
MSVC follows a stricter, more traditional C++ model regarding intrinsics.
In MSVC, __m512i is defined in the header files (<immintrin.h>) as a struct or union (e.g., typedef struct __m512i { ... } __m512i). To the MSVC compiler, it is essentially a user-defined data type, not a fundamental language primitive like int or float.
Standard C++ does not define what `|` means for a user-defined struct.
MSVC does not have the same "Vector Extensions" that automatically apply operators to these structs.
When you write `a | b` in MSVC, the compiler looks for a definition of `operator|` for the __m512i struct. Since the standard headers don't provide one, the compiler throws an error.
You must use the explicit intrinsic function provided by Intel/MSVC: _mm512_or_si512(a, b).
To get the nice syntax `(a | b)` in MSVC, you have to manually "teach" the compiler what `|` means by defining the `operator|` overload yourself.
* Update README.md with build instructions for Windows
Current README lacks any guide for Windows users, whereas build process on that platform is quite compicated
* Update build.md with instruction about clang-cl.exe
Brings step-by-step build instruction for Windows
* Apply suggestions from code review
Co-authored-by: Kawrakow <iwankawrakow@gmail.com>
* Polish build.md for Windows usage
Added example of use for Windows
* Apply suggestions from code review
---------
Co-authored-by: Kawrakow <iwankawrakow@gmail.com>
Adam Caldwell
Kawrakow
Nexes the Elder
StrikeOner
hksdpc255
mcm007
dungquixote42
firecoperana
SneedwareInc
saood06
Marcel Coetzee
mullecofo