* Add GLM-5.2/DeepSeek-V3.2 DSA lightning indexer (batch-local, single-seq prefill)
Implements the sparse top-k "lightning indexer" attention for LLM_ARCH_GLM_DSA
in build_deepseek2_layer_attention (ik's deepseek2 graph).
What it does (per layer, gated on model.arch==GLM_DSA && indexer_attn_q_b):
- indexer_q = indexer_attn_q_b(q_lora latent), split rope(64)/nope(64), NEOX-rope
the pe part, concat. indexer_k = indexer_attn_k(attn_norm out), LayerNorm w/ bias,
same rope/concat (single key head, MQA).
- scores = relu(indexer_k . indexer_q), scaled per-head weights (indexer_proj),
summed over heads, + base causal mask, then ggml_top_k(min(top_k, n_tokens)).
- sparse mask: ggml_fill(-inf) -> ggml_set_rows(0) at top_k positions -> + causal,
used in the soft_max_ext attention path (-mla 1 -fa 0) instead of KQ_mask.
Simplifications (intentional, proven sound):
- Batch-local: no indexer KV-cache. Indexer keys are the current batch tokens.
- Walsh-Hadamard transform omitted: orthonormal rotation, (Hq).(Hk)==q.k, no score change.
Validation (GLM-5.2-UD-IQ2_M, 3x P100, -mla 1 -fa 0):
- Compiles clean (CUDA sm_60); loads and runs.
- c512 -b512 (n_seq=1) PPL = 2.7760, byte-identical to dense baseline (indexer
disabled) = 2.7760, all 8 chunks match -> indexer is an exact no-op when
top_k>=n_tokens. Proves correctness-preservation.
- 3105-token prompt completion (top_k=2048 < 3105 -> indexer ACTIVELY masks):
prompt-eval produces coherent, accurate continuation, identical to dense for the
prompt+early-gen tokens. No NaN/crash. Confirms the masking path works in prefill.
Known limitations (documented follow-ups, NOT handled):
- Single-sequence prefill only. Multi-sequence batches (n_seq>1, e.g. perplexity
default n_batch>n_ctx) and kv_head>0 (decode) break the batch-local key->slot
mapping. n_seq>1 -> NaN (use n_batch==n_ctx). Decode (kv_head>0): each generated
token sees only itself as an indexer key, so generation degenerates into repetition
after the prompt (dense A/B stays coherent) -- this is the decode-cache stub, the
documented next step.
- Flash-attn path (-fa 1, F16 mask) still uses dense KQ_mask (soft_max path only).
- Decode indexer KV-cache + Hadamard cached-K storage not implemented.
Runtime gate: DSA_INDEXER_DISABLE=1 falls back to dense attention (for A/B).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: decode-correct via persistent indexer-K cache
Make the lightning-indexer correct for DECODE (not just prefill). Previously the
indexer was batch-local, so a generated token only scored against itself and
generation degenerated. Now the indexer keys are cached across the full context.
Changes
- llama_kv_cache: add per-layer indexer-key cache `kr_l` [indexer_head_size, kv_size]
(F16, MQA single head), allocated alongside the MLA latent cache for GLM_DSA.
- build_deepseek2_dsa_indexer: write the batch's (Hadamard-rotated) indexer keys to
kr_l at kv_head, read back the full [128, n_kv] cached keys, and score the indexer
queries against ALL past keys. Returns the full descending argsort of the scores.
- Walsh-Hadamard rotation of indexer q/k (cparams.dsa_indexer_hadamard, default on;
filled in llama_set_inputs). Score-preserving; improves cached-K F16 precision.
- build_deepseek2_dsa_sparse_mask: rank-based full-coverage scatter (write a 0/-BIG
penalty into EVERY key slot keyed by rank) instead of partial set_rows into a -inf
fill — the CUDA in-place set_rows does not preserve an un-written base, which had
corrupted decode when n_kv > top_k.
- Attention-sink force-inclusion (DSA_SINK, default 1): boost the first key(s) so the
sink always survives top-k. The IQ2_M-quantized indexer under-ranks the sink, and
masking it collapsed decode; with the boost, top_k=2048 over n_kv>2048 stays coherent.
ggml backend fixes (needed by the indexer)
- CUDA argsort: report unsupported when padded ncols > 1024 (one-thread-per-column
bitonic launch limit) so the scheduler falls back to the CPU argsort. Fixes
"invalid configuration argument" for top_k over a large n_kv.
- CUDA cpy/dup: support I32 -> I32 (top_k index copies / cross-backend moves).
Validation (GLM-5.2-UD-IQ2_M, 3xP100 + --cpu-moe, -mla 1 -fa 0)
- c512 PPL = 2.0743, byte-identical to dense (all 8 chunks): no-op path exact.
- Short-context decode (300 tok): coherent, identical to dense.
- Long-context decode (2521-tok prompt, n_kv>top_k, real masking of ~474 keys,
120+ tok generated): coherent with the sink boost; dense A/B also coherent.
Gated behind arch==GLM_DSA + indexer tensors + kr_l cache; DSA_INDEXER_DISABLE=1
forces dense. Remaining: FA path still uses the dense KQ_mask; multi-sequence
(n_seq>1) batches; deepseek32 arch wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: wire sparse mask into the flash-attention path (-fa 1)
The DSA sparse top-k mask is now applied on the -fa 1 path (our serving config),
not just -fa 0 soft_max. c512 PPL on -fa 1 = 2.0743, byte-identical to dense
(no regression, indexer no-op exact at n_kv <= top_k). Gated arch==GLM_DSA with
DSA_INDEXER_DISABLE escape; -fa 0 path unchanged.
Long-context -fa 1 decode coherence (n_kv > top_k, mask actually biting) validation
is still running at commit time; the FA mask reuses the same full-coverage scatter
proven coherent on the -fa 0 decode path, so it should hold, but confirm before
relying on long-context -fa 1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: UPDATE 4 — MLA-FA fix merged, FA path validated, multi-seq characterized
Document the re-validation after cherry-picking the MLA-FA vec-decode fix (5f18dcc0):
- FA path is ALIVE. Long-ctx -fa 1 decode (2521-tok prompt > top_k, mask actively
biting) is now COHERENT at -mla 1 and -mla 3, vs the pre-fix degeneration into
"0.0.0.0..." repetition. Matches dense (DSA_INDEXER_DISABLE) and -fa 0 controls.
- c512 -fa 1 PPL: indexer-ON == dense == 2.0854, byte-identical all 8 chunks (exact
no-op when n_kv <= top_k; no regression). The 2.0743->2.0854 shift is the MLA-FA
fix changing V accumulation, not an indexer artifact (ON==dense proves it).
- Indexer is feature-complete + validated for single-seq prefill+decode on both
-fa 0 and -fa 1, at -mla 1 and -mla 3 (the R740 serving target).
Remaining PR gaps, characterized honestly:
- Multi-seq (n_seq>1) with active mask is BROKEN (n_seq=2 c4096 PPL 62.6 vs dense
multi-seq 2.54 and single-seq indexer 3.05). No NaN/crash anymore. Root cause:
the indexer uses a single scalar kv_head/n_kv for the whole ubatch; multi-seq
needs per-sequence cache writes + per-sequence top-k. Fix deferred (structural).
- deepseek32 arch: N/A in this fork. DSA lives entirely under LLM_ARCH_GLM_DSA;
there is no LLM_ARCH_DEEPSEEK32 enum. Documented the steps to add one if a real
deepseek32 GGUF is ever served.
Also commit DSA_REFERENCE.md (verbatim mainline deepseek32/glm-dsa source, the port
reference), trimmed of a stray agent-handoff footer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: per-sequence attention sink — fix multi-seq (n_seq>1)
UPDATE 5. The DSA lightning indexer was numerically broken for multi-sequence
batches once the top-k mask bites (n_kv > top_k): c4096 n_seq=2 PPL 62.6 vs
dense 2.54, while single-seq was fine. Root cause: the attention-sink
force-include boosted the GLOBAL key range [0, n_sink) by +1e20, which only
protects sequence 0's sink. With several sequences packed contiguously into one
ubatch (seq 0 at cells [0,n0), seq 1 at [n0,n1), ...), every non-first
sequence's sink lives at cell n0.. (not cell 0), got no boost, and was dropped
from top-k once the mask bites — collapsing that sequence (chunk[2]=61.2 while
chunk[1]=2.33).
The cache write and score/argsort were already per-sequence correct: tokens are
placed contiguously like the main K cache, and the base KQ_mask (filled from
kv_self.cells[i].has_seq_id) already drives cross-seq keys to -inf before
argsort. Only the sink was anchored at the wrong (global) cell.
Fix: replace the global arange sink boost with a per-graph input tensor
inp_dsa_sink {n_kv, n_tokens} (F32), filled on the CPU in llama_set_inputs from
kv_self.cells exactly like the KQ_mask:
inp_dsa_sink[j,i] = 1e20 iff cell[i].pos in [0,n_sink) AND
cell[i].has_seq_id(seq_of_query_j), else 0
so each query force-includes only its OWN sequence's sink. For a single
contiguous sequence from pos 0 this is exactly the old "cell index < n_sink"
set with the same magnitude, so n_seq==1 is byte-identical.
Validation (3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1, wikitext-2):
- c4096 n_seq=2 indexer chunk[2]: 61.2 -> 3.07 (== single-seq 3.05).
- c2048 topk=1024 (mask bites): n_seq=4 == n_seq=1 chunk-for-chunk
(2.5005/2.6080/2.7759/3.1137 vs .../3.1138) -> multi-seq is numerically
identical to processing each sequence alone.
- c512 n_seq=1 indexer ON == dense, all 4 chunks byte-identical (no regression).
n_seq=4 at full c4096 (n_kv=16384) OOMs the P100 compute buffer (capacity, not
correctness; n_seq=4 proven correct at c2048/n_kv=8192).
GLM-5.2 DSA indexer is now sequence-correct for n_seq>=1, prefill+decode,
soft_max+FA, -mla 1/-mla 3. Fully general and PR-ready.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: UPDATE 6 — serving-correctness (kr_l maintained across shift/defrag/seq-ops; per-seq sink on first-present pos)
An adversarial review found the indexer was proven on the perplexity path but
not the serving path: the persistent indexer-K cache kr_l was written/read but
never *maintained* by the KV-cache mutators, and the attention sink anchored on
absolute pos<n_sink (wrong after multi-turn seq_rm). This closes those gaps and
pins down what is actually reachable on the MLA model.
kr_l maintenance:
- build_k_shift (llama-build-context.cpp): rotate the indexer keys by the same
per-cell delta as the main K. The cached key is H*concat(RoPE(k_pe,pos),k_nope),
so un-Hadamard (H sym/orthonormal => H*H=I) -> RoPE-delta the pe sub-block ->
re-Hadamard. Exact because GLM-DSA has no rope-scaling metadata (ext_factor=0,
attn_factor=1, freq_scale=1), so NEOX RoPE is pure/composable. Params mirror the
forward indexer RoPE exactly (rope_factors=nullptr); no DEEPSEEK2 yarn-shift leak.
Non-in-place (cont->rope->concat->re-Had->cpy), no aliasing. K-shift Hadamard
input filled in llama_set_k_shift with the identical Sylvester construction.
- build_defrag: kr_l row-move mirrors the k_l move (defrag never changes pos, so
no re-RoPE). max_moves divisor 6->9 *n_layer when the indexer cache is present.
- seq_rm/seq_cp/seq_keep are metadata-only (verified) so kr_l rows stay matched to
cells; seq_add/seq_div set has_shift and route through K-shift. No seq-op change.
Per-seq sink (llama.cpp llama_set_inputs): anchor on each sequence's FIRST PRESENT
pos (min present pos over the scored n_kv span), not absolute pos<n_sink. After
multi-turn seq_rm drops a sequence's early tokens its earliest survivor has
pos>=n_sink; the absolute test would protect nothing. Fresh seq at pos 0 => min=0
=> byte-identical to the old behaviour.
Serving-shift finding (the whole point): a RoPE context-shift on this model is
REFUSED BY THE ENGINE. get_can_shift() returns false for all MLA models
(is_mla_model() includes GLM_DSA); llama_kv_cache_update returns 1 ->
"main : failed to eval". Reproduced AND isolated with a dense control
(DSA_INDEXER_DISABLE=1): dense fails identically at the same token. The failure is
pre-existing MLA engine behaviour, independent of the indexer. On the MLA path the
shift never happens, so the indexer's kr_l can never desync via K-shift; the
build_k_shift kr_l block is correct-and-dormant (documented loudly in code).
Validation (3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1, GGML_CUDA_NO_PINNED=1,
numactl --interleave=all, wikitext-2):
- No regression: c512 n_seq=1 indexer ON == dense == 2.1957 +/- 0.12031,
byte-identical all 4 chunks (2.2770/2.8741/2.3956/2.1957).
- Multi-seq: c4096 n_seq=2 chunk[1]=2.33 chunk[2]=3.07 healthy (== UPDATE 5;
per-seq sink change did not regress).
- Serving shift: engine-refused for MLA, dense control fails identically.
- Independent adversarial review: GO, no correctness defect in the diff.
- Build clean (llama-cli, llama-perplexity, sm_60).
Comments updated (build_deepseek2.cpp): multi-seq+FA no longer limitations; sink
description matches per-seq min-pos anchoring; BIG=1e30 masks on both soft_max and
FA paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: UPDATE 7 — FIX latent graph-reuse cache-fixup omission for the kr_l indexer cache
update_cache_copies() re-points the K/V cache writes to the current kv_head whenever a
compute graph is REUSED (can_reuse_graph reuses iff kv_self.n == prev->n_kv). The persistent
indexer-key cache write (kr_l) is a separate ggml_cpy whose destination view bakes kv_head at
graph-build time, and it was NEVER registered for that fixup. Under FA the cache pads to 256,
so consecutive single-token decode ubatches share the same padded n_kv and the graph IS reused;
without the fixup the kr_l write keeps landing in the first ubatch's slot and later ubatches
never populate their own recent index-key cells (those cells stay at the alloc-zeroed 0.0).
Structurally identical to the MiniMax MSA bug (fork commit 133d14c9).
Fix (mirrors the K/V cache_copies fixup, same shape as MSA 133d14c9):
- llama-context.h: new std::vector<CacheCopy> dsa_cache_copies.
- llama.cpp ctor: resize dsa_cache_copies to n_layer (null entries -> no-op when DSA off).
- build_deepseek2.cpp: register the kr_l ggml_cpy as dsa_cache_copies[il] = {kr_cpy, kr->nb[1]}.
- llama.cpp update_cache_copies(): re-point each registered cpy view_offs = kv_head*step and
patch src[1]->data/data, exactly like K/V, with the c.cpy->view_src == kv_self.kr_l[il]
(+ null/op) guard the MSA fix omitted. soft_max / non-DSA paths byte-identical.
Validation (GLM-5.2-UD-IQ2_M, 3x P100 -ngl 99 --cpu-moe -t 32, NO_PINNED, P2P-disable patch
re-applied to get a working multi-GPU baseline — see UPDATE 7.3; that patch was lost in the
upstream rebase and is required separately):
- c512 -fa1 -mla3 indexer ON: 2.1983 (== prior baseline; build healthy).
- Long-ctx FA decode, 2735-tok recall prompt, -mla3 -fa1 temp0, reuse ON (default): coherent,
correct deep-context recall ("Dr. Mariana Velasquez ... Daniel Okonkwo") on BOTH the fixed and
the unfixed binary.
- ub128 PPL -fa1 -mla3 reuse ON, unfixed: 1.7239/1.8211/2.1888/2.4517, healthy (no inflation).
Honest scope: the bug is real in code but LATENT for GLM-DSA at its configured top_k=2048
(permissive selection keeps the genuinely-attended recent blocks even when reuse leaves some
recent index-key cells stale), unlike MSA's tighter top-k where it inflated PPL ~2x. The fix is
correct and prevents the latent corruption from biting at any tighter top_k / longer ctx /
future serving config. The pre-P2P-patch "nan" seen at ub128 was P2P corruption, not this bug.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-DSA: convert sparse-attention control from env vars to CLI args (off by default)
Implements ikawrakow's direction from discussion #2040: the DSA sparse
indexer must be controllable via command-line argument (not environment
variables), and must be OFF by default for now.
Control surface, before -> after:
DSA_INDEXER_DISABLE (env, inverted: on-by-default) -> --dsa / -dsa
(cparams.dsa, default false; opt-in, dense-by-default)
DSA_TOPK_OVERRIDE (env) -> --dsa-top-k N / -dsatk N
(cparams.dsa_top_k, default -1 == model's configured indexer_top_k)
DSA_HADAMARD_DISABLE, DSA_SINK (env) -> kept as DEBUG-ONLY env
knobs (clearly commented; no CLI surface, not system on/off controls)
Plumbing mirrors existing boolean/int feature flags (-mla, -khad):
include/llama.h llama_context_params {bool dsa; int dsa_top_k;}
src/llama.cpp default_params (false / -1); cparams assignment
src/llama-cparams.h llama_cparams {bool dsa=false; int dsa_top_k=-1;}
common/common.h gpt_params {bool dsa=false; int dsa_top_k=-1;}
common/common.cpp arg parse + help text + cparams copy
src/graphs/build_deepseek2.cpp gate now checks cparams.dsa instead of
getenv; top-k override reads cparams.dsa_top_k. Stays arch-gated to
LLM_ARCH_GLM_DSA. When --dsa is off (default) the indexer function is
never called -> existing dense MLA path, byte-identical to no-feature.
Validation (GLM-5.2-UD-IQ2_M, 3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1,
wikitext-2, 4 chunks @ c2560):
--dsa OFF (default, dense): PPL 2.4151 (graph nodes 4166)
--dsa ON, default top_k=2048: PPL 2.4697 (graph nodes 8846)
--dsa ON, --dsa-top-k 1024: PPL 3.5107
Off-by-default runs the dense path; ON activates the indexer (node count
jumps, PPL shifts as the top-k mask bites once n_kv > top_k). No env var
is consulted for the primary on/off or the top-k knob.
Graph-parallel (-sm graph) interaction (the item ikawrakow flagged):
Under -sm graph the MLA layers are TP-split (wo->extra) and route to
build_deepseek2_tp_attention(), which contains NO indexer code. So --dsa
is silently a NO-OP under -sm graph: it does not error or crash, it runs
dense. Empirically, --dsa --dsa-top-k 1024 under -sm graph gives
PPL 2.4308 (chunks 1.6967/1.7906/2.1664/2.4308) -- the dense baseline
(2.4151), NOT the DSA top_k=1024 numbers (3.5107). The 0.016 delta is
f16 TP-reduce numerics, not DSA. Conclusion: DSA "works under deepseek2"
only on the non-TP (layer) path; serving DSA with -sm graph would require
wiring the indexer into the TP attention path (or a dedicated DSA arch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-DSA: warn that --dsa is inactive under -sm graph/attn (TP path runs dense MLA)
The DSA lightning indexer is built only in the layer-mode (non-TP) attention
path. Under -sm graph / -sm attn the tensor-parallel attention path has no
indexer, so --dsa would silently run dense MLA. Emit a clear one-time
LLAMA_LOG_WARN at context creation instead of degrading silently.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-DSA: drop in-tree dev reference docs from the PR branch
DSA_REFERENCE.md and the R740 progress note are development scratch, not
part of the submission. Remove them so the PR diff is code-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-DSA: fix CPU-only crashes in the sparse-attention path
PR #2045 adds GLM-DSA sparse attention but was validated on CUDA (--cpu-moe).
A CPU-only build (-ngl 0 --dsa) crashes in four spots where the CUDA backend
tolerates something the CPU backend does not. These make GLM-5.2 --dsa run
coherently on CPU; with --dsa off they are no-ops (DSA CPU path only).
1. set_rows into an F32 dest segfaults (ggml.c set_rows_f32):
type_traits[F32].from_float is NULL, so the DSA sparse-mask scatter calls a
NULL fn (segfault at ip=0). memcpy when the dest is F32. CUDA has a real F32
set_rows path, so this only bit the CPU build.
2. ggml_add(F32 score, F16 mask) aborts on CPU (build_deepseek2_dsa_indexer and
build_deepseek2_dsa_sparse_mask): under -fa 1 the dense KQ_mask is F16 and CPU
add only accepts F32+F16 when src0 is F16. Cast the causal mask view to F32.
CUDA's add accepts the mixed types.
3. dsa_fa_mask dim-1 concat must be F32 on CPU (build_deepseek2_dsa_fa_mask):
CPU ggml_concat only supports F16 along dim 0; do the row (dim-1) concat in
F32 then cast the result to F16. CUDA supports the F16 dim-1 concat.
4. indexer k_norm epsilon is 0 -> ggml_norm aborts (llama-hparams.cpp): the
lightning-indexer k_norm is a non-RMS LayerNorm using f_norm_eps, but the
GLM-DSA GGUF only carries the RMS eps so f_norm_eps stays 0
(GGML_ASSERT(eps > 0)). Mirror the RMS eps. CUDA's norm doesn't assert on eps=0.
Validated: GLM-5.2 UD-Q4_K_M, single-socket Xeon w7-2475X, CPU-only (-ngl 0 --dsa)
- coherent at 49K+ ctx, correct 30K needle retrieval, prefill flat with length
(~32 tok/s, the O(L) DSA signature) vs the dense build's O(L^2) decline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* DSA: loop over attention heads + use builtin Hadamard
* DSA: ggml_blend
* DSA: remove a bunch of unnecessary ggml_cont
* DSA: fix CUDA blend - but something is still wrong
* DSA: use ggml_top_k instead of ggml_argsort when FA is ON
* CUDA: add CUB based argsort
* DSA: avoid graph leaves
* Various
* GLM-5.2 DSA: IndexShare (shared layers reuse full-layer top-k)
GLM-5.2's indexer_types marks 21 'full' layers that compute their own
lightning-indexer top-k and 57 'shared' layers that reuse the previous
full layer's top-k. This port computed an independent top-k on every
layer, which mis-selects keys on the 57 shared layers (the transformers
reference sets indexer=None on shared layers and reuses prev_topk).
Shared layers now reuse the most-recent full layer's selection. Full/
shared map derived from the config rule (full iff il<=1 or il%4==2),
which reproduces indexer_types exactly; loader can later override from
GGUF metadata. Built on #2063's tree; head-loop/ggml_hadamard/ggml_blend/
argsort/FA-mask unchanged.
4K PPL (unsloth IQ2_M, top_k 2048, CPU): DSA-on 3.1922 -> 2.7111, dense
2.6972 (~97% of the gap). top_k>=n_kv reproduces dense exactly. Single-
seq and 4x8 parallel decode coherent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply suggestion from @ikawrakow
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: mgkwill <168222+mgkwill@users.noreply.github.com>
Co-authored-by: Kawrakow <iwankawrakow@gmail.com>
* Support MiMo DFlash draft conversion
* Fix MiMo2 DFlash capture row pruning
* Fix MiMo DFlash draft RoPE and value scale
* Honor partial_rotary_factor in DFlash draft RoPE dim count
The draft set rope.dimension_count to the full head_dim (128), ignoring the
MiMo DFlash draft's partial_rotary_factor=0.5. The correct count is
head_dim*partial_rotary_factor=64; the remaining dims are NoPE. With the full
head_dim the upper half of each head receives position rotation it was never
trained for, which roughly halves draft acceptance on code (~26% -> ~60% once
corrected). RoPE base (5e6) and value scale (0.612) were already correct.
* Filter weight-map shard discovery to files that exist
get_model_part_names_from_weight_map() returned shard names straight from the
index weight_map without checking they exist. A model dir with a stale
model.safetensors.index.json but no safetensors shards would then set
is_safetensors=True and skip the pytorch_model*.bin fallback, failing later when
opening the missing files. Filter to shards present on disk so a stale index
falls through to the other weight formats.
* DFlash: store backbone_rotary_base in dedicated GGUF key
backbone_rotary_base (the target model's RoPE theta used when encoding
context K/V) was written to rope.freq_base, clobbering the draft
model's own rope_theta. For MiMo this swapped 10000 → 5000000 in the
draft attention path.
Fix: write backbone_rotary_base to a dedicated dflash.backbone_rotary_base
GGUF key and read it into hparams.dflash_backbone_rotary_base. In
build_dflash_kv_cache, use target_freq_base (the new hparam when set,
falling back to freq_base) for the context-K RoPE call. The draft model's
own rope.freq_base is now set correctly from rope_theta.
Existing MiMo DFlash GGUFs must be reconverted.
---------
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
* DFlash: bound intra-block draft tokens to the SWA window
The SWA mask builder applied the sliding-window distance check only to the
cross-context section; the intra-block draft-token loop masked causal-only,
so a draft token could attend to earlier block tokens beyond n_swa. Apply
the same window bound ((j - block_k) < swa_window) in both the F16 and F32
paths so it matches the cross-context section.
Behavior-neutral for dense models: the SWA mask tensor is only allocated
when the model has SWA layers (build_dflash.cpp needs_swa_mask gate), so
for dense targets the changed block is unreachable.
* DFlash: enable sliding-window attention for draft models
DFlash drafts can be trained with sliding-window attention for long context,
but the runtime ignored it: the draft loader never read the window keys and the
converter never emitted them, so SWA-trained drafts always ran full-attention.
Enable it end to end and fix the dormant SWA graph path it exposes:
- convert_hf_to_gguf.py (DFlashDraftModel): emit attention.sliding_window + an
all-layers sliding_window_pattern when the source config sets use_sliding_window.
- llama-hparams.cpp (LLM_ARCH_DFLASH_DRAFT): read sliding_window + pattern into
n_swa / swa_layers.
- build_dflash.cpp + llama-dflash.cpp: the SWA mask path had never run; an all-SWA
draft turned the full kq_mask into a dead graph node the scheduler never backs
with a buffer, then the input-set wrote it unconditionally (GGML_ASSERT buf!=NULL).
Create + set each mask only when a layer uses it; derive mask dims from whichever
mask is live. Dense/mixed drafts are byte-identical.
Validated on gemma-4-26B-A4B at long context (cross_ctx 8176 > window 2048): no
crash, no short-context regression, SWA-on recovers long-context draft acceptance.
* DFlash: derive draft SWA pattern from layer_types
The converter emitted an all-layers SWA pattern ([True]*n_layers). The z-lab
DFlash drafts are sliding-window on every layer except a final full-attention
(global) layer, so this ran that global layer as sliding-window and clipped its
long-context view. Read layer_types and emit the matching per-layer pattern
(sliding_attention -> True), falling back to all-SWA only when layer_types is
absent.
---------
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
* wip: port MTP architecture
Ports the Multi-Token Prediction (MTP) architecture to the older `llama.cpp` codebase used by `ikllama`.
Changes include:
- Updating `llama_batch` to support `mtp_params`.
- Modifying `llama_decode_internal` (and `encode`) to handle MTP operations (Warmup, Update, Draft).
- Adding public APIs for MTP state management (`llama_set_draft_input_hidden_state`).
- Adapting the embedding extraction logic to skip MTP update passes.
* Refactors `server_slot` to support generic speculative decoding (MTP or Draft Model).
* core: enable hybrid outputs (logits + embeddings) for MTP support
* fix(mtp): correct KV-cache slot finding for updates
* fix(mtp): persist hidden states to prevent context corruption during drafting
* refactor(mtp): clean unused code
* fix(mtp): update server to new functions name
* fix(mtp): fix graph and save hidden state
* mtp: refactor integration, context params and kv cache search
* mtp: fix hidden state extraction and speculative acceptance flow
* server: fix MTP warmup for long prompts and reset token buffer
* llama: refactor MTP operation state to context parameters
* server: fix n_past calculation in MTP acceptance
* llama: fix mtp enable flags
* speculative: refactor MTP to use common_speculative interface
* context: remove unused signatures
* clip: fix deprecated enum-enum conversion warning
* common: fix format string crash in help message
* context: fix mtp activation logic
* llamat: always use the extracted embedding
* llama: get all embeddings to kv cache
* llama: revert logit to not run mtp for not supported arch
* llama: allocate all the n_outputs for MTP
* wip
* server-context: get only the last embedding for hidden state
* ggml-backend: fix array of bounds in debug build
* server-context: run mt kv update to each prompt batch
* revert segmentation fault fixes
* glm-mtp(feat): optimize graph embedding and recursive drafting
* glm5-mtp(feat): add glm 5 mtp logic
* glm-mtp: standardize the MTP graph
* glm 5 mtp: apply post-layer cvec
* glm 5 mtp: mark head as mandatory
* get normed embeddings for glm 5
* Fix GLM5 MTP
* GLM5 MTP: just reuse the layer attention implementation
* Make MTP work with split mode graph
---------
Co-authored-by: samuel <samueloliveira32df@gmail.com>
* Gemma4: WIP
* Gemma4: WIP - runs with totally wrong results
* Gemma4: WIP - add CPU 512, 512 FA
* Gemma4: WIP
It gives a meaningful response in llama-cli, but PPL is still much too
high. Is this due to tokenizer issues?
* Gemma4: this works
I had forgotten the softcap on the final output.
* Remove log
* Gemma4: WIP E4B/E2B
* Gemma4: Q4B/E2B appear to work now
* gemma4: tokenizer fixes
* wip: port MTP architecture
Ports the Multi-Token Prediction (MTP) architecture to the older `llama.cpp` codebase used by `ikllama`.
Changes include:
- Updating `llama_batch` to support `mtp_params`.
- Modifying `llama_decode_internal` (and `encode`) to handle MTP operations (Warmup, Update, Draft).
- Adding public APIs for MTP state management (`llama_set_draft_input_hidden_state`).
- Adapting the embedding extraction logic to skip MTP update passes.
* Refactors `server_slot` to support generic speculative decoding (MTP or Draft Model).
* core: enable hybrid outputs (logits + embeddings) for MTP support
* fix(mtp): correct KV-cache slot finding for updates
* fix(mtp): persist hidden states to prevent context corruption during drafting
* refactor(mtp): clean unused code
* fix(mtp): update server to new functions name
* fix(mtp): fix graph and save hidden state
* mtp: refactor integration, context params and kv cache search
* mtp: fix hidden state extraction and speculative acceptance flow
* server: fix MTP warmup for long prompts and reset token buffer
* llama: refactor MTP operation state to context parameters
* server: fix n_past calculation in MTP acceptance
* llama: fix mtp enable flags
* speculative: refactor MTP to use common_speculative interface
* context: remove unused signatures
* clip: fix deprecated enum-enum conversion warning
* common: fix format string crash in help message
* context: fix mtp activation logic
* qwen3next: add architecture support and recurrent-state fixes
* qwen3next: optimize broadcast sub and single-seq ssm conv
* cuda: build MoE row mapping on device in mul_mat_id
* cuda: add guarded multi-seq fast path for ssm_conv
* docs: update qwen3next perf report for cuda MoE/SSM tuning
* cuda: reduce qwen3next moe/ssm sync overhead and refresh eval
* qwen3next: split cpu/cuda eval builds and tune PP scheduling
* qwen3next: harden seq-state flow and support optional dense FFN layers
* qwen3next: trim delta-net graph overhead in chunking path
* qwen3next: remove redundant v_conv cont in delta path
* qwen3next: avoid extra cont on linear attention output
* qwen3next: drop redundant cont before recurrent state flatten
* qwen3next: keep recurrent state in 4d layout through delta path
* qwen3next: add fused delta-net op and wire model path
* tests: add backend-op coverage for ggml_delta_net
* qwen3next: add runtime switch for fused delta-net path
* docs: refresh qwen3next perf review and benchmark matrix
* qwen3next: default fused delta-net off and document quality checks
* qwen3next: add decode-only fused delta mode
* qwen3next: make fused delta safe by default and fix fused tensor layout
* qwen3next: warn when forcing fused decode mode
* qwen3next: add fused-delta regression runner script
* qwen3next: integrate fused regression into eval harness
* qwen3next: clean up chunked delta-net shape handling
* qwen3next: add absolute sanity guards to fused regression
* qwen3next: add unified regression runner script
* qwen3next: disable flash-attn for cpu-only contexts
* docs: reconcile qwen3next status and remaining upstream gaps
* common: add qwen3next fused-delta runtime flag
* cuda: add qwen3next delta-net kernel dispatch override
* docs: update qwen3next quality and serving baseline findings
* qwen3next: keep fused delta on safe path and remove PR artifacts
* qwen3next: align autoregressive delta-net decode layout
* Revert "qwen3next: align autoregressive delta-net decode layout"
This reverts commit 9241164a5ea9e032a2456fbf2dd0bf798b264fd7.
* cuda: port solve-tri fast-paths for qwen3next delta-net
* qwen3next: add fused-delta runtime flag and drop env toggle
* qwen3next: make fused delta single-flag and default on
* Account for GPU arch differences
* Revert "cuda: build MoE row mapping on device in mul_mat_id"
This reverts commit 89e9ecfa840b04e88699ab3803eb732cd78727f9.
* qwen3next: drop non-essential MoE scheduling and split heuristics
* qwen3next: avoid generic ggml_sub broadcast changes
* llama: restore only_active_experts log message
* Remove unnecessary hacks, disable fusion for now.
* qwen3next: port hybrid recurrent state memory semantics
* qwen3next: clean up recurrent state slot plumbing
* qwen3next: fix hybrid V-cache layout plumbing
* qwen3next: guard recurrent state slots against kv capacity
* qwen3next: persist recurrent state in session data
- serialize/restore qwen3next cache.s_l in state/session paths\n- bump session and sequence-state file versions for format change\n- fallback to single-token chunking for mixed repeated seq_id batches
* qwen3next: drop unused fused-delta builder path
- remove dead build_delta_net_fused lambda\n- remove unused llm_build_context::fused_delta member
* qwen3next: remove unused fused-delta CLI/context plumbing
- drop -fd/-no-fd options and related YAML dump field\n- remove fused_delta fields from public/internal context params\n- remove fused_delta assignment and logging in context init
* ggml: remove unused DELTA_NET operator stack
* Missing include
* Reorder ops/unary ops
So we don't change again the enum values of the mul mat ops
* Minor
* Discard unnecessary changes in llama-build-context.cpp
* Minor
* Revert "Discard unnecessary changes in llama-build-context.cpp"
This reverts commit edadb80ed68c4c0831e9c22609a9a3af19be9735.
* Increase GGML_SCHED_MAX_SPLITS - required for larger u-batches
* Fix CPU concat in the TG case: 7.25 -> 10.5 t/s for Qwen3Next
* Fix CPU sum_rows: 10.5 -> 13.6 t/s for Qwen3Next
It was single-threaded and was taking ~25% of the computation time
during TG. It is now down to 2%.
Strangely enough, I measure 13.6 t/s with llama-bench, but if I
let the model give me an actual response with llama-cli, I get close
to 17 t/s.
* Fix CPU scale: 13.6 -> 16.7 t/s for Qwen3Next
For Qwen3Next there is a scale op on a largish tensor (548k elements)
that has a single row for TG, so was done in a single thread.
We now simply use blocks of 1024 elements.
* Optimize CPU mul: 16.7 -> 17.6 t/s for Qwen3Next
* CPU: fuse transpose -> cont -> sum_rows -> transpos: 17.6 -> 23.1 t/s for Qwen3Next
* Optimize CPU repeat: 176 -> 200 t/s for Qwen3Next PP-512
* Multithreading for OP_SUB
* Don't commit with timing trace on
* Multithread neg and sigmoid
* Be able to turn on/off fusion more easily (CPU)
* Name the mul_mat ops so we know where the time goes
* WIP
* Much better PP on CUDA
* CUDA: fuse transpose -> cont -> sum_rows -> transpose
Needs non-coontiguous variant of sum_rows.
On the CPU this gave 30+% improvement in TG performance,
on CUDA ist is disapointing 6-7%. I guess, this is because
Georgi's cont CPU implementation was so bad that skipping
it made such a big difference.
* CUDA: faster mul for special case relevant for Qwen3Next
Worth 1% in TG
* Fix CPU OP_CONT
---------
Co-authored-by: yurko <yurko@local>
Co-authored-by: Yurko <yurko@example.com>
Co-authored-by: yurko <yurko@pop-os.tail5a1a6b.ts.net>
Co-authored-by: Yurko Hoshko <YurkoHoshko@users.noreply.github.com>
* WIP
* This works but is slow
* Turn off the up / gate clamps for now
* OK we need the clamping
* Fuse the clamp (CUDA)
* Fuse the clamp (CPU)
* WIP
* Be able to use merged q, k, v
* Be able to use merged up/gate experts
* Fuse the clamp (CUDA mmvq)
* Mimo-2 support
* Fix bug for head sizes not being the same
It still does not solve the Mimo-2 quantized cache issue.
* Fix quantized cache
* Minor
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
* model : Granite docling + Idefics3 preprocessing (SmolVLM) (#16206)
* feat: Add granite-docling conversion using trillion pretokenizer
Branch: gabe-l-hart/GraniteDocling
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Add granite-docling vocab pre enum
Branch: gabe-l-hart/GraniteDocling
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Use granite-docling pre
Branch: gabe-l-hart/GraniteDocling
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Add clip_is_idefics3
Branch: gabe-l-hart/GraniteDocling
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Allow multi-token boundary sequences for image templating
Branch: gabe-l-hart/GraniteDocling
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Add tiling support for idefices3 in clip.cpp
This should likely be moved into llava_uhd::get_slice_instructions, but for
now this avoids disrupting the logic there.
Branch: gabe-l-hart/GraniteDocling
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Partial support for full templating for idefics3 in mtmd
There are still errors encoding some of the image chunks, but the token
sequence now matches transformers _almost_ perfectly, except for the double
newline before the global image which shows up as two consecutive newline
tokens instead of a single double-newline token. I think this is happening
because the blocks are tokenized separately then concatenated.
Branch: gabe-l-hart/GraniteDocling
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Fully working image preprocessing for idefics3 w/ resize and slicing
Branch: gabe-l-hart/GraniteDocling
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* feat: Parse the preprocessor config's longest side and add it to the mmproj hparams
Branch: GraniteDocling
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Use the longest side instead of size * scale_factor
For Granite Docling, these come out to the same value, but that was just a
conicidence.
Branch: GraniteDocling
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* fix: Allow batch encoding and remove clip_is_idefics3
Branch: GraniteDocling
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* refactor: Remove unnecessary conditionals for empty token vectors
Branch: GraniteDocling
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* refactor: Use image_manipulation util
Branch: GraniteDocling
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
* add test model
---------
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
# Conflicts:
# convert_hf_to_gguf.py
# convert_hf_to_gguf_update.py
# gguf-py/gguf/constants.py
# gguf-py/gguf/gguf_writer.py
# src/llama-vocab.cpp
# src/llama-vocab.h
* mtmd : support home-cooked Mistral Small Omni (#14928)
* model : add LightOnOCR-1B model (#16764)
* model : add LightOnOCR-1B model
* add test
# Conflicts:
# convert_hf_to_gguf.py
# gguf-py/gguf/constants.py
* mtmd : fix idefics3 preprocessing (#16806)
* mtmd : fix idefics3 preprocessing
* disable granite test
* fix test for granite
* model: Add support for CogVLM model (#15002)
* Added GGUF mappings for CogVLM model
* Add tensor mapping for CogVLM visual encoder
* Add CogVLM to conversion script, no vision part yet
* Added CogVLM vision model to conversion script
* Add graph for CogVLM CLIP model
* Add graph for CogVLM
* Fixes for CogVLM. Now compiles.
* Model now runs
* Fixes for cogvlm graph
* Account for graph context change after rebase
* Changes for whitespace
* Changes in convert script according to comments
* Switch CogVLM LLM graph to merged QKV tensor
* Use rope_type variable instead of direct definition
* Change CogVLM CLIP encoder to use SWIGLU
* Switch CogVLM CLIP to use merged QKV
* Apply rebase edits and remove ggml_cont call that is now unnecessary
* clean up
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
# Conflicts:
# convert_hf_to_gguf.py
# examples/mtmd/clip.cpp
# gguf-py/gguf/constants.py
# gguf-py/gguf/tensor_mapping.py
# src/llama-arch.cpp
# src/llama-arch.h
# src/llama-model.cpp
# src/llama-model.h
* mtmd: refactor preprocessing + support max/min pixels (#16878)
* mtmd: refactor preprocessing + support max/min pixels
* fix mlp type
* implement mix/max pixels
* improve hparams
* better image preproc for qwen
* fix
* fix out of bound composite
* fix (2)
* fix token calculation
* get_merge_kernel_size()
* fix llama4 and lfm2
* gonna fix them all
* use simple resize for qwen
* qwen: increase min tokens
* no resize if dst size == src size
* restore to initial min/max tokens value for qwen
# Conflicts:
# examples/mtmd/clip.cpp
* clip : use FA (#16837)
* clip : use FA
* cont : add warning about unsupported ops
* implement "auto" mode for clip flash attn
* clip : print more detailed op support info during warmup
* cont : remove obsolete comment [no ci]
* improve debugging message
* trailing space
* metal : remove stray return
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
* model: add Janus Pro for image understanding (#16906)
* Add support for Janus Pro
* Update gguf-py/gguf/tensor_mapping.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* Update gguf-py/gguf/tensor_mapping.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* Address reviewer suggestions
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* Add JANUS_PRO constant
* Update clip model handling
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
* Update tools/mtmd/clip.cpp
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
* Refactor JANUS_PRO handling in clip.cpp
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
* Update tools/mtmd/clip.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* em whitespace
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
# Conflicts:
# convert_hf_to_gguf.py
# gguf-py/gguf/constants.py
# gguf-py/gguf/tensor_mapping.py
* mtmd: pad mask for qwen2.5vl (#16954)
* mtmd: pad mask for qwen2.5vl
* improve
* mtmd: add --image-min/max-tokens (#16921)
* mtmd: improve struct initialization (#16981)
* mtmd: allow QwenVL to process larger image by default (#17020)
* Disable flash attention
* mtmd : fix embedding size for image input (#17123)
* mtmd: fix patch_size initialized to random value in audio models (#17128)
* mtmd: fix patch_size initialized to random value in audio models
* add default hparams
* add llama_model_n_embd_inp
* Fix load qwen3 vl
Change batch size
* Add description
* Fix cli build error
---------
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: Tianyue-Zhao <zhaotianyue@outlook.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Zhiyong Wang <85110830+ravenouse@users.noreply.github.com>
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
Co-authored-by: firecoperana <firecoperana>
* Fixing Gigachat support
* Gigachat: CUDA FA (needs 192 x 192 for MLA = 3)
* Gigachat: CPU FA (needs 192 x 192 for MLA = 3)
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
* llama_model and llama_hparams
* llama_build_context
Surprisingly small reduction in llama.cpp compile time given
the reduction in LOCs (22k -> 14k)
* LLM_TN
llama.cpp compilation: 50 s -> 33 s
* llama_quantize
* arch names
* All graph building is now in llm-build-context.cpp
* hparams loading
llama.cpp is now just 9300 LOC, but still takes 32 seconds to compile.
* We are now at 6 seconds to build the src folder
* load -> create
We are not actually loading the tensors, but just creating them.
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
mb8565
Joel Farthing
empty-quiver
Kawrakow
Samuel Oliveira Alves
Jun Yamog
SamuelOliveirads
saood06
firecoperana
Thireus ☠