common/chat, server: refactor, move all conversion functions to common, add tests (#20690)
jinja : remove unused header (#22310)
common : fix jinja warnings with clang 21 (#22313)
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
chat: fix handling of space in reasoning markers (#22353)
* chat: fix handling of space in reasoning markers
common : re-arm reasoning budget after DONE on new <think> (#22323)
common : determine generation prompt using longest common prefix (#22657)
common/autoparser: fixes for newline handling / forced tool calls (#22654)
* chat/autoparser: the fixes
* Move optspace() to chat-peg-parser, comment out server tests invalidated due to content now allowed with forced tool calls.
* Trim whitespace on apply instead
common/chat : preserve media markers for typed-content templates (#22634)
common : revert reasoning budget +inf logit bias (#22740)
common : do not wrap raw strings in schema parser for tagged parsers (#22827)
common : enable streaming JSON argument values (#23173)
* common : remove atomic from json arguments
* common : remove parsing logic on JSON arguments
common : do not pass prompt tokens to reasoning budget sampler (#22488)
reasoning-budget: clone should do a deep-copy (#23095)
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
If a slot is reused for a standard completion (`/v1/completions`) after
being used for a chat completion (`/v1/chat/completions`), the previous
chat's PEG parser would remain active in the slot's parameters. This
caused standard text completions to throw on the raw text.
Add `-tm` / `--threads-mtmd` to control CPU thread count used during
multimodal image/audio processing (mmproj encoding), separate from the
main LLM thread count.
This allows running the LLM on GPU with minimal CPU threads (e.g. `-t 1`)
to reduce sync overhead, while using many threads (e.g. `-tm 16`) for
CPU-bound mmproj encoding with `--no-mmproj-offload`.
Fallback chain when `-tm` is not specified:
1. `--threads-batch` (-tb) — multimodal encoding is a batch/prefill-like
operation, so it makes sense to track with batch thread count
2. `--threads` (-t) — final default
Works with both mtmd-cli and llama-server.
AI: ubergarm/Qwen3.6-27B-GGUF MTP IQ4_KS 15.113 GiB (4.752 BPW) + pi.dev
The get_batch_ubatch() function unconditionally inflated n_batch and
n_ubatch whenever --mmproj was specified, regardless of whether the
mmproj model actually ran on the GPU. This boosted batch size applies
to both the main context and the MTP draft context, since
params_base.speculative.cparams_dft is derived from
common_context_params_to_llama(params_base).
When mmproj runs on CPU (--no-mmproj-offload), this batch inflation
is unnecessary for mmproj itself (CPU compute is sized by image
dimensions independently), but it still inflates the MTP compute buffer
proportionally. For large images (e.g. --image-max-tokens 4096), the
MTP compute buffer ballooned to ~2020 MiB and triggered an OOM even
though the mmproj model was fully on CPU and should have saved VRAM.
Restrict the batch inflation to !params.mmproj.path.empty() &&
params.mmproj_use_gpu so it only triggers when mmproj actually occupies
GPU memory. When mmproj runs on CPU, the existing per-chunk decode
splitting in mtmd_helper_decode_image_chunk_impl handles large images
correctly with the default batch size.
AI: ubergarm/Qwen3.6-27B-GGUF MTP IQ4_KS 15.113 GiB (4.752 BPW) + pi.dev
* Avoid copying the per-step SSM state (CUDA)
* Avoid copying the per-step SSM state (CPU)
* Allocate only what is necessary for per-step SSM state
* Cleanup
* Use AVX version VNNI intrinsic when AVX512VNNI not available.
* remove changes under HAVE_FANCY_SIMD
---------
Co-authored-by: XZiar <xziar@xziar.xziar>
This patch addresses two latent bugs in examples/speculative/speculative.cpp
that prevent llama-speculative.exe from running on greedy sampling
(temp=0) or producing rejection-sampling output (temp>0):
1. Line 191: `params.sparams.grammar = { COMMON_GRAMMAR_TYPE_NONE, "" };`
invokes `common_grammar(type, grammar)` which asserts
`type != NONE || !grammar.empty()`. Both conditions fail with the
intended-to-be-empty grammar, so every speculative run hits a hard
`GGML_ASSERT` in common/sampling.h:63 immediately after model load.
Fix: default-construct via `common_grammar{}` to bypass the
field-init constructor.
2. Lines 293-294: `GGML_ASSERT(dist_tgt.sorted)` and
`GGML_ASSERT(dist_dft.sorted)` fire whenever the draft sampler does
not set the .sorted flag (which is most modern sampler paths).
Comment them out — the next ~10 lines re-sort both distributions
by id explicitly, so the assertion is incorrect anyway.
Fix: replace the asserts with an explanatory comment.
After both fixes, `llama-speculative.exe` runs to completion. The
acceptance-rate measurement at temp=0 still looks suspicious (0%
across same-family draft/target pairs), but that is a different
issue out of scope for this PR.
Tested on Qwen3-0.6B-IQ4_XS drafting Qwen3-1.7B-IQ4_XS, both base
models from `bartowski/Qwen_Qwen3-*-GGUF` on Windows + ik_llama.cpp
build at HEAD of windows-mingw-default-win10 (which is itself a
follow-up to PR #1755).
The default of 0x602 (Windows 8) causes a build failure on any toolchain
where _WIN32_WINNT propagates into vendored cpp-httplib (notably MinGW with
the bundled w64devkit GCC). cpp-httplib's httplib.h has, for some time
now, contained:
#ifdef _WIN32
#if defined(_WIN32_WINNT) && _WIN32_WINNT < 0x0A00
#error "cpp-httplib doesn't support Windows 8 or lower. Please use
Windows 10 or later."
#endif
#endif
so the entire llama-server target fails to compile on Windows + MinGW
unless the user passes -DGGML_WIN_VER=0x0A00 manually.
Bumping the default to 0x0A00 (Windows 10) keeps Windows 8 reachable for
anyone who explicitly requests it (-DGGML_WIN_VER=0x602) while letting the
default Windows + MinGW build succeed end-to-end. Windows 8 / 8.1 reached
end of support in January 2023, and Windows 10 is a strict superset of the
Win8 surface used elsewhere (PrefetchVirtualMemory etc.), so this is
strictly additive on the API side.
Verified by building with w64devkit 2.8.0 (gcc 16.1.0) on Windows 11
without any -DGGML_WIN_VER override: all 266 ninja targets link cleanly,
including bin/llama-server.exe, and llama-cli runs Qwen3-4B-Thinking-2507
IQ4_XS at ~6.2 tok/s with q8_0 KV at 4096 context.
* Add Turing and Ampere (A100) GGML to docker build file
At the moment, the docker file for image builds do not build for CUDA architectures below 8.6, and ik_llama.cpp specifies support for architectures Turing and above, this PR sets the CUDA architecture list to include the architecture for Turing (7.5) and A100 (8.0)
* Remove 80 because few ppl have A100s and it does seem like many cuda arches cause issues for build
* switch to 86-real and 89-real with 75, 80, 90 using virtual ptx jit
* nvm, even adding 90-virtual causes linker error
---------
Co-authored-by: Codex <codex@local>
Adds a short note in README's "Build for CPU" section pointing to the
AVX-512 build flags reference in docs/build.md (added by #1729).
The vanilla `cmake -B build -DGGML_NATIVE=ON` example shown right above
silently falls back to the AVX2 path on AMD Zen4 / Intel Sapphire
Rapids+ hardware; users hitting "my Zen4 build is slow" tend to look at
the README first, so a single-paragraph cross-reference here saves them
from having to dig through docs/ to find the right knob.
No content moved — README still has its own short example, the new
paragraph just points at the deeper reference.
firecoperana
Kawrakow
gapeleon
Samuel Oliveira Alves
Jun Yamog
Marian M.
ubergarm
Forkoz
Lingfeng Ren
XZiar
Alex
joelfarthing
Zhekun Hu
Henrik Berglund
dungquixote42
Andrew Moryakov