* DS4 optimizations (part 2)
* This is slightly better
* Another minor tweak
* Increase max. number of graph splitinputs to 64
Else with DS4 we can trun into an assert for specific offload
situations with more than one GPU.
* Adding ds4_comp op with CPU implementation
* ds4_comp on CUDA
* ds4_comp: ratio = 4 specialization
Surprisingly small performance gain
* Also handle HCA via ds4_comp
But much smaller gain, if any.
* Delete commented out stuff
* Remove the [(size_t) il] noise
* Minor
* Fix quantized cache
Adds ggml_latent_attn_prefix_ext / ggml_latent_attn_indexed_ext: MLA
latent-cache attention with an always-visible learned K/V prefix
(openPangu's 128 param_sink rows), joint softmax over [prefix | cache],
reading the raw F32/F16/Q8_0 latent cache directly. CUDA implementation
plus a scalar CPU reference that pins the op's semantics; the CPU
backend reports support truthfully, and openPangu adopts the op only on
a non-CPU backend as builder policy.
openPangu routes its dense/SWA/MTP full-span attention and the gathered
DSA path through the op, capability-gated per layer on the attention
output projection's scheduled backend, with the latent cache required
resident on that same backend (--no-kv-offload keeps the unfused
chain); any layer whose backend cannot run the candidate keeps the
exact unfused chain.
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
* indexer_topk: fix quantized q8_1 scratch sizing on CUDA
The quantized-K path under-sized its q8_1 scratch buffer: it allocated q->ne[1]*max_rows blocks using the unpadded head dim, but quantize_mmq_q8_1_cuda writes q_padded/QK8_1 blocks per row and must process all q->ne[1]*nrows rows. Size the buffer by (q_padded/QK8_1) blocks x (q->ne[1]*max_rows) rows and pass the full q->ne[1]*nrows row count so the scratch cannot be overrun and every query row is quantized.
CUDA graph identity: INDEXER_TOPK dispatches a source-type-specialized kernel (dense F16 vs quantized cache, F32 vs F16 mask). Source addresses alone do not identify the captured kernel, so snapshot each source type in ggml_graph_node_properties and force re-capture when an INDEXER_TOPK source type changes, preventing a reused graph from replaying the wrong kernel variant when sources are reallocated at the same address.
CPU backend: report INDEXER_TOPK support via iqk_indexer_topk_supported (guarded by GGML_USE_IQK_MULMAT) so the scheduler places the node on a backend that can run it. Harden the scheduler's pass-5 node-assignment check from assert to GGML_ASSERT so a node no backend supports fails as a defined abort under NDEBUG instead of indexing sched->backends[-1].
* indexer_topk: drop CPU capability predicate
* indexer_topk: fall back when unsupported
* cuda: drop speculative INDEXER_TOPK graph type matching
---------
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
* initial map to load deepseek 4 arch
* wip
* wip: match graph build and attn logic for dpv4
* wip: Enhance DeepSeek-V4 architecture with new tensor types and sqrtsoftplus gating function
* Update DeepSeek-V4 to support raw key indexing with read/write indices
* fix mismatch in attn_raw
* Enable FA with CSA/HCA
* Fix logit mismatch with FA path
* Clean traces and logs for debug
* Refactor DSV4 tensor handling for MTP execution and improve raw context management
* Refactor DeepSeek4 tensor operations: replace manual weighted sum and post-processing with new helper functions
* Share mHC pre-projection and fix packed DSV4 writes
* DSV4: add shared top-k selection and improve mask handling
* Fix DSV4 c2048 view stride and duplicate loader instantiation
* Reuse shared RMS normalization in DSV4 graph
* Replace DSV4 indexer rotation with shared Hadamard
* Share CSA visibility mask with DSV4 LID
* dsv4: document dependency ordering and reset state
* Remove DSV4 zero-dependency graph shim
* Fix DSV4 packed stream execution
* Remove DSV4 l_out backend override
* Enable DSV4 quantized K-only cache
* Revert "Enable DSV4 quantized K-only cache"
This reverts commit 04f9b425321f62ba60e16d1bea2f8de714cfe855.
* Fix DSV4 quantized cache accounting
* Fail closed on unsupported DSV4 cache lifecycle operations
* Various optimizations
* llama: fix GGML_METAL=ON build - missing ggml-metal.h include in llama-dflash.cpp (#2134)
llama-dflash.cpp calls ggml_backend_is_metal() and
ggml_backend_metal_set_n_cb() inside an #ifdef GGML_USE_METAL block but
never includes ggml-metal.h, so any Metal-enabled build fails to
compile. Add the same guarded include llama.cpp already uses.
* New op: ggml_sum_rows_ext (#2132)
* Add ggml_sum_rows_ext
* openPangu: use ggml_sum_rows_ext also in mhc_post
* openPangu: use ggml_sum_rows_ext also in mhc_tail
* Minor
* Reuse shared inverse RoPE operation for DSV4
* Reuse maintainer CUDA concat implementation
* WIP
* hc_pre
* hc_post
* Remove unnecessary mask manipulations
* WIP
* Take into account swiglu limits
* Turn on fused indexer by default
* Give names to mat mul results
* More named ops
* dsv4: do not uselessly copy the KV cache
+20% TG at 32k tokens
* mask_to_index and make CPU FA work with that
* Much better CPU-only, CUDA still not functional
* Better CPU TG
I'm now at 9.7 t/s for zero context and 6.5 t/s for context of 32k.
PP is 120 t/s for short context and 101 t/s at 32k.
* Even better CPU TG
I'm now at 8.1 t/s for context of 32k tokens.
* Turn off DSA on CUDA for now
* Fix CUDA DSA
* Remove again the unnecessary softmax result buffer
* Experiments
* Various
* More named ops
* Forgot to uncomment
---------
Co-authored-by: samuel <samueloliveira32df@gmail.com>
Co-authored-by: hchengit <95317477+hchengit@users.noreply.github.com>
The Metal backend lacked GGML_ROPE_TYPE_MROPE/IMROPE support, so models
whose GGUF carries rope_sections (e.g. Qwen 3.5 hybrids) could not run
fully offloaded on Apple Silicon.
Add rope_multi_f32/f16 kernels with section-based position handling
(t/h/w/e blocks, 4 position ids per token), imrope's interleaved section
selection, and the corresponding dispatch in ggml-metal.m. Vision-mode
mrope is not implemented and is asserted out explicitly.
Validated on M2: kernel output matches the CPU backend, and Qwen 3.5-9B
(Q4_K_M, -ngl 99) WikiText-2 perplexity over 145 chunks lands within
0.006 of the same model's CPU baseline.
Faster fp32-class replacement for the tile_f32 flash-attention inner path on P100
(sm_60). Bit-identical fp32 arithmetic to the un-retiled kernel (same-top 99.28% =
the float-reorder floor; QK differs only in accumulation order, ~1 ulp; P.V
bit-identical), restructured via half2 K/V shared-memory staging that both cuts
shared memory and leans the inner loop.
Measured +4-9% vs the un-retiled fp32 tile kernel, back-to-back. The speedup is a
COMPOUND of two effects the staging produces together (shares not isolated):
(1) occupancy: smem 36992 -> 28800 B/block admits 2 blocks/SM where the un-retiled
kernel fits only 1 (2*28800=57600<=65536; 80 regs would allow 3, smem is the
ceiling); a genuine 1->2 gain, corroborated by a cross-family perf panel;
(2) a leaner inner loop: ~2x fewer QK-loop smem loads, the dropped score round-trip,
one fewer barrier.
__launch_bounds__(...,2) only PINS the target the smem reduction already reaches (a
(...,1) bound compiles bit-identically), so the directive is not itself a lever, and a
3rd block does not help (cost-curve + an implemented regs-80->124 attempt confirm smem
caps occupancy at 2).
Validated on P100 as the drop-in for the carve-out path (pr-p100-fp16). Reviewed by a
6-model Claude council + a cross-family cloud perf panel; the panel corrected an earlier
'not occupancy' framing to this compound one.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On P100 (GP100, sm_60) the fp16 vec kernel used for decode (batch<=8)
accumulates the online-softmax denominator and the P*V product in fp16,
flipping ~3-4% of decode top-1 tokens vs an all-fp32 reference
(llama.cpp#25593). Decode is memory-bandwidth-bound on P100, so routing
sm_60 decode to the in-tree vec_f32 kernel is free (tg128 ~identical).
Gated on cc == CC_PASCAL && Q->ne[1] <= 8 (decode only) inside the
!fp16_mma_available block, so the prefill tile_f16 path, the D=256 prefill
vec path, and fast_fp16_available() are untouched, and the
is_pascal_mla_absorbed_decode early-return (MLA) is unaffected.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ggml_graph_node_has_matching_properties exempted every GGML_OP_CPY node from the source-address check, for both operands. The exemption is needed for KV-cache write copies, whose destination advances each step through the indirect-destination path. It also skips a CPY whose read source (src[0]) moves between graph replays while the surrounding subgraph stays shape-stable, so the captured kernel replays against a stale source pointer and reads the previous step's bytes.
Keep the exemption for the destination operand (src[1]) only, and compare a CPY's read source (src[0]) like any other node. Stable-source copies are unaffected and force no additional re-captures; a CPY whose read source genuinely moves now re-captures instead of reading stale memory.
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
* WIP: indexer_topk on CUDA
* Forgot these
* WIP
* WIP
* This seems to work
* Minor
* Fix bug. Fix suggested by @sayap using GLM-5.2
* GLM-DSA: much better PP long context performance (CUDA)
* DSA: Better way to build the attention mask
* WIP: indexer_topk on CUDA
* Forgot these
* WIP
* WIP
* This seems to work
* Minor
* Fix bug. Fix suggested by @sayap using GLM-5.2
* GLM-DSA: much better PP long context performance (CUDA)
* ggml: add fused sinkhorn op (eps + output-layout params); use it for openPangu mHC
* openpangu: call ggml_sinkhorn directly from mhc_post
---------
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
* IQK AVX2: Replace MM256_SET_M128I(x, x) identity broadcasts with _mm256_broadcastsi128_si256
## Summary
Replace all 132 instances of the identity-broadcast pattern
`MM256_SET_M128I(x, x)` with `_mm256_broadcastsi128_si256(x)` across
8 files in ggml/src/iqk/.
## Correctness
The transformation is bit-exact. The `MM256_SET_M128I(a, b)` macro
expands to:
_mm256_insertf128_si256(_mm256_castsi128_si256(b), a, 1)
which places `a` in the high 128-bit lane and `b` in the low lane.
When `a == b == x`, the result is x replicated to both lanes:
[x_lo: x_hi] = {x, x}
`_mm256_broadcastsi128_si256(x)` produces the identical register state:
it copies the 128-bit input to both lanes in a single micro-op.
Perplexity was verified identical on llama-3.2-1b-Q8_0 and
google-gemma-3-4b-Q4_0-IQ4_XS before and after the change.
The transformation is a pure intrinsic substitution with
zero semantic difference.
## Performance
### Micro-architecture analysis
The old pattern compiles to:
vinsertf128 ymm, ymm, xmm, 1 -- 3 uops, port 5, 3-cycle latency
The new pattern compiles to:
vbroadcasti128 ymm, xmm -- 1 uop, port 5, 1-cycle latency
Both instructions execute on port 5 (Intel), but vbroadcasti128:
- Uses 1/3 the dispatch slots (fewer pipeline stalls)
- Has 3x better latency (1 vs 3 cycles)
- Is not lane-crossing (no bypass delay between 128-bit halves)
### Measured results
#### Test 1: llama-3.2-1b-Q8_0, 2048 ctx, -b 128 -ub 128
Compiler | Metric | Before | After | Change
--------------|--------|---------|---------|-------
MSVC 19.44 | PP t/s | 596.89 | 604.57 | +1.29% (noise imo)
MSVC 19.44 | TG t/s | 55.13 | 55.72 | +1.07% (noise)
Clang 19.1.5 | PP t/s | 645.72 | 700.93 | +8.55% (systematic gain)
Clang 19.1.5 | TG t/s | 54.26 | 54.06 | -0.37% (noise)
#### Test 2: google-gemma-3-4b-Q4_0-IQ4_XS, 4096 ctx, -b 512
Compiler | Metric | Before | After | Change
--------------|----------|----------|----------|-------
Clang 19.1.5 | PP t/s | 262.40 | 314.79 | +19.96% (systematic gain)
Clang 19.1.5 | TG t/s | 30.78 | 32.32 | +5.00% (noise, TG oscilates between 31.5 and 33 t/s before / after)
Clang 19.1.5 | Total ms | 48881 | 44696 | -8.56%
Both tests ran on Intel Core Ultra 265K, 18 threads, flash_attn=1
The IQ4_XS result shows a dramatic PP improvement (+20%) because this
quantization format uses significantly more identity broadcasts in its
dequantization path (lookup-table expansion, scale duplication). The
compressed 4-bit representation requires more setup per block, making
the broadcast-to-256 step a measurable bottleneck that the 1-uop
vbroadcasti128 eliminates.
### Why these 132 instances matter
Every dequantization path (IQ2_XXS through IQ6_K, Q4_0 through Q8_1,
MXFP4) starts by broadcasting a 128-bit lookup table or scale vector
to 256 bits. These are in the inner loop of every quantization format's
dot-product kernel. Reducing each broadcast from 3 uops to 1 uop
cumulatively reduces port-5 pressure across the entire dequant
pipeline.
## Scope
This change touches only the identity-broadcast case (both MM256_SET_M128I
arguments are identical).
* IQK AVX2: Introduce MM256_SET1_M128I(x) wrapper for identity-broadcast pattern
Per Ikawrakow's review: replace direct _mm256_broadcastsi128_si256(x) with
a new macro MM256_SET1_M128I(x) wrapping the intrinsic, so that if the
broadcast turns out harmful on some CPU, only the macro definition needs
changing, not 132 call sites.
#define MM256_SET1_M128I(x) _mm256_broadcastsi128_si256(x)
The 132 identity-broadcast MM256_SET_M128I(x, x) call sites across 8 files
now use MM256_SET1_M128I(x) instead of the raw intrinsic.
* Add --prefetch-experts to stream mmap'd MoE experts into page cache
* Drop fds, fault experts in with MADV_POPULATE_READ instead of pread
* Remove stale note about pread workers
* Move MoE prefetch behind ggml_backend_prefetch_* wrappers
* Cleanup stale comments
* Add --prefetch-experts-threads, drop GGML_MOE_PREFETCH_THREADS env var
When building with clang-cl (MSVC + Clang), the CMake MSVC branch defined
__AVXVNNI__ as a preprocessor macro alongside /arch:AVX2, but clang-cl
requires the actual -mavxvnni target feature flag to enable AVX-VNNI
codegen. Without it, clang-cl refused to inline _mm256_dpbusd_avx_epi32
and _mm256_dpwssd_avx_epi32 into functions compiled under /arch:AVX2,
causing 'requires target feature avxvnni' errors in:
- ggml-quants.c (mul_sum_us8_pairs_float)
- iqk_gemm_iquants.cpp (mul_mat_iq3_xxs_r4_q8_k)
- iqk_gemm_kquants.cpp (mul_mat_q3_k_r4_q8_k)
- iqk_gemm_legacy_quants.cpp (dot, accum_q4_0_quants, operator())
Fix: Detect clang-cl via CMAKE_CXX_COMPILER_ID STREQUAL 'Clang' and
append -mavxvnni to ARCH_FLAGS instead of manual __AVXVNNI__ define.
Also add missing GGML_AVXVNNI handling for the non-MSVC (GCC/Clang on
Linux) branch, passing -mavxvnni as expected.
* CUDA: use the mask tensor stride (nb31) in the tile FA kernels, not ne11
The tile flash-attention kernels (no-tensor-core GPUs, e.g. Pascal/sm_60) indexed
the KQ mask using ne11 (= K->ne[1]) as the row stride. The SWA windowing in
ggml_cuda_flash_attn_ext (the n_swa branch) re-points K/V/mask to the last nton
tokens, setting ne11 = nton while the mask keeps its original row stride nb31.
Indexing by ne11 then reads across mask rows and yields NaN once the window slice
engages (context past nton). Use the mask's own stride nb31/sizeof(half); in the
non-sliced case this equals ne11, so it is a no-op there. Matches how the vec
kernel already computes its mask offset, and how mainline llama.cpp fixed the same
latent bug in its unified tile kernel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* CUDA: apply attention sinks in the tile FA kernels (gpt-oss)
The tile flash-attention kernels took a sinks argument but never applied it, so
gpt-oss (which has a per-head sink logit) got a wrong softmax denominator with
-fa 1 while the -fa 0 path (ggml_soft_max_add_sinks) matched CPU. Apply the sink
after the KV loop, mirroring the vec kernel: the sink joins the running max and
adds exp(sink - max) to the denominator once, rescaling kqsum and VKQ. Only ip==0
adds it so it is counted once across the parallel_blocks KV split (the epilogue
writes the sink-inclusive max/denominator into dst_meta before the combine). The
per-head index is blockIdx.y (the same index the slope uses). Guarded by non-null
sinks, so non-sink models are unaffected. This is the tile-kernel equivalent of
mainline llama.cpp PR #15178; the wmma kernel is left untouched (no Turing/Volta
hardware to validate here, but the same defect likely applies).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On GPUs without FP16 tensor cores (Pascal / sm_60, e.g. Tesla P100) MLA
flash-attention decode falls back to the CPU. The !fp16_mma_available path
routes decode to the f16 vector kernel, whose is_supported check requires
K == V head sizes; MLA's absorbed head sizes are 576/512 (asymmetric), so it is
rejected and attention runs on the CPU. With --cpu-moe that recomputes the full
MLA attention on the CPU every decoded token, which dominates decode at long
context.
Route Pascal MLA decode (Q->ne[1] <= 8 && K == 576 && V == 512) to the f32
vector kernel and enable that kernel for the 576/512 case, including Q8_0 KV.
Scope: decode only (batch <= 8). Prefill (batch > 8) and -fa 0 are untouched;
tensor-core GPUs never reach this branch. Aligned head sizes are byte-identical
(the asymmetric/Q8_0 work folds to a no-op at compile time), so no other model
or configuration is affected.
- fattn.cu: route 576/512 decode to vec_f32 in the !fp16_mma dispatch and its
is_supported mirror.
- fattn-vec-f32.cu/.cuh: accept + instantiate 576/512 (F16 and Q8_0); fix latent
issues exposed by the first asymmetric/large-head use (KQ-row granularity uses
FATTN_KQ_STRIDE not Dv; guard the dst store to tid < Dv; guard the softmax exp
on the KV tail; size Q_i32 by ceil; only convert K/V to F16 when the type is
F16). All are no-ops for the previously-exercised symmetric cases.
- fattn-vec-f32.cuh / fattn-vec-common.cuh: guard the Q8_0 ragged tail (Dk=576 is
144 int32 lanes = 4.5 warps) with the ragged-dim idiom; compile-time-constant
for aligned head dims, so it folds away.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The scalar fast-path in ggml_cuda_op_mul routes to ggml_cuda_op_scale_tensor,
which reads src0 and writes dst as flat contiguous buffers. With a non-contiguous
src0 (for example a row-gapped view) this ignored the per-row strides: only the
first row was correct and later rows read the wrong memory. The CPU backend
respects the strides, so the two backends diverged.
Guard the fast-path on contiguous src0 and dst; non-contiguous inputs now fall
through to the general bin_bcast path, which honours the strides.
Add a non-contiguous test_bin_bcast variant covering both the scalar (scale) and
vector (general) paths.
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
* Add GLM-5.2/DeepSeek-V3.2 DSA lightning indexer (batch-local, single-seq prefill)
Implements the sparse top-k "lightning indexer" attention for LLM_ARCH_GLM_DSA
in build_deepseek2_layer_attention (ik's deepseek2 graph).
What it does (per layer, gated on model.arch==GLM_DSA && indexer_attn_q_b):
- indexer_q = indexer_attn_q_b(q_lora latent), split rope(64)/nope(64), NEOX-rope
the pe part, concat. indexer_k = indexer_attn_k(attn_norm out), LayerNorm w/ bias,
same rope/concat (single key head, MQA).
- scores = relu(indexer_k . indexer_q), scaled per-head weights (indexer_proj),
summed over heads, + base causal mask, then ggml_top_k(min(top_k, n_tokens)).
- sparse mask: ggml_fill(-inf) -> ggml_set_rows(0) at top_k positions -> + causal,
used in the soft_max_ext attention path (-mla 1 -fa 0) instead of KQ_mask.
Simplifications (intentional, proven sound):
- Batch-local: no indexer KV-cache. Indexer keys are the current batch tokens.
- Walsh-Hadamard transform omitted: orthonormal rotation, (Hq).(Hk)==q.k, no score change.
Validation (GLM-5.2-UD-IQ2_M, 3x P100, -mla 1 -fa 0):
- Compiles clean (CUDA sm_60); loads and runs.
- c512 -b512 (n_seq=1) PPL = 2.7760, byte-identical to dense baseline (indexer
disabled) = 2.7760, all 8 chunks match -> indexer is an exact no-op when
top_k>=n_tokens. Proves correctness-preservation.
- 3105-token prompt completion (top_k=2048 < 3105 -> indexer ACTIVELY masks):
prompt-eval produces coherent, accurate continuation, identical to dense for the
prompt+early-gen tokens. No NaN/crash. Confirms the masking path works in prefill.
Known limitations (documented follow-ups, NOT handled):
- Single-sequence prefill only. Multi-sequence batches (n_seq>1, e.g. perplexity
default n_batch>n_ctx) and kv_head>0 (decode) break the batch-local key->slot
mapping. n_seq>1 -> NaN (use n_batch==n_ctx). Decode (kv_head>0): each generated
token sees only itself as an indexer key, so generation degenerates into repetition
after the prompt (dense A/B stays coherent) -- this is the decode-cache stub, the
documented next step.
- Flash-attn path (-fa 1, F16 mask) still uses dense KQ_mask (soft_max path only).
- Decode indexer KV-cache + Hadamard cached-K storage not implemented.
Runtime gate: DSA_INDEXER_DISABLE=1 falls back to dense attention (for A/B).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: decode-correct via persistent indexer-K cache
Make the lightning-indexer correct for DECODE (not just prefill). Previously the
indexer was batch-local, so a generated token only scored against itself and
generation degenerated. Now the indexer keys are cached across the full context.
Changes
- llama_kv_cache: add per-layer indexer-key cache `kr_l` [indexer_head_size, kv_size]
(F16, MQA single head), allocated alongside the MLA latent cache for GLM_DSA.
- build_deepseek2_dsa_indexer: write the batch's (Hadamard-rotated) indexer keys to
kr_l at kv_head, read back the full [128, n_kv] cached keys, and score the indexer
queries against ALL past keys. Returns the full descending argsort of the scores.
- Walsh-Hadamard rotation of indexer q/k (cparams.dsa_indexer_hadamard, default on;
filled in llama_set_inputs). Score-preserving; improves cached-K F16 precision.
- build_deepseek2_dsa_sparse_mask: rank-based full-coverage scatter (write a 0/-BIG
penalty into EVERY key slot keyed by rank) instead of partial set_rows into a -inf
fill — the CUDA in-place set_rows does not preserve an un-written base, which had
corrupted decode when n_kv > top_k.
- Attention-sink force-inclusion (DSA_SINK, default 1): boost the first key(s) so the
sink always survives top-k. The IQ2_M-quantized indexer under-ranks the sink, and
masking it collapsed decode; with the boost, top_k=2048 over n_kv>2048 stays coherent.
ggml backend fixes (needed by the indexer)
- CUDA argsort: report unsupported when padded ncols > 1024 (one-thread-per-column
bitonic launch limit) so the scheduler falls back to the CPU argsort. Fixes
"invalid configuration argument" for top_k over a large n_kv.
- CUDA cpy/dup: support I32 -> I32 (top_k index copies / cross-backend moves).
Validation (GLM-5.2-UD-IQ2_M, 3xP100 + --cpu-moe, -mla 1 -fa 0)
- c512 PPL = 2.0743, byte-identical to dense (all 8 chunks): no-op path exact.
- Short-context decode (300 tok): coherent, identical to dense.
- Long-context decode (2521-tok prompt, n_kv>top_k, real masking of ~474 keys,
120+ tok generated): coherent with the sink boost; dense A/B also coherent.
Gated behind arch==GLM_DSA + indexer tensors + kr_l cache; DSA_INDEXER_DISABLE=1
forces dense. Remaining: FA path still uses the dense KQ_mask; multi-sequence
(n_seq>1) batches; deepseek32 arch wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: wire sparse mask into the flash-attention path (-fa 1)
The DSA sparse top-k mask is now applied on the -fa 1 path (our serving config),
not just -fa 0 soft_max. c512 PPL on -fa 1 = 2.0743, byte-identical to dense
(no regression, indexer no-op exact at n_kv <= top_k). Gated arch==GLM_DSA with
DSA_INDEXER_DISABLE escape; -fa 0 path unchanged.
Long-context -fa 1 decode coherence (n_kv > top_k, mask actually biting) validation
is still running at commit time; the FA mask reuses the same full-coverage scatter
proven coherent on the -fa 0 decode path, so it should hold, but confirm before
relying on long-context -fa 1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: UPDATE 4 — MLA-FA fix merged, FA path validated, multi-seq characterized
Document the re-validation after cherry-picking the MLA-FA vec-decode fix (5f18dcc0):
- FA path is ALIVE. Long-ctx -fa 1 decode (2521-tok prompt > top_k, mask actively
biting) is now COHERENT at -mla 1 and -mla 3, vs the pre-fix degeneration into
"0.0.0.0..." repetition. Matches dense (DSA_INDEXER_DISABLE) and -fa 0 controls.
- c512 -fa 1 PPL: indexer-ON == dense == 2.0854, byte-identical all 8 chunks (exact
no-op when n_kv <= top_k; no regression). The 2.0743->2.0854 shift is the MLA-FA
fix changing V accumulation, not an indexer artifact (ON==dense proves it).
- Indexer is feature-complete + validated for single-seq prefill+decode on both
-fa 0 and -fa 1, at -mla 1 and -mla 3 (the R740 serving target).
Remaining PR gaps, characterized honestly:
- Multi-seq (n_seq>1) with active mask is BROKEN (n_seq=2 c4096 PPL 62.6 vs dense
multi-seq 2.54 and single-seq indexer 3.05). No NaN/crash anymore. Root cause:
the indexer uses a single scalar kv_head/n_kv for the whole ubatch; multi-seq
needs per-sequence cache writes + per-sequence top-k. Fix deferred (structural).
- deepseek32 arch: N/A in this fork. DSA lives entirely under LLM_ARCH_GLM_DSA;
there is no LLM_ARCH_DEEPSEEK32 enum. Documented the steps to add one if a real
deepseek32 GGUF is ever served.
Also commit DSA_REFERENCE.md (verbatim mainline deepseek32/glm-dsa source, the port
reference), trimmed of a stray agent-handoff footer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: per-sequence attention sink — fix multi-seq (n_seq>1)
UPDATE 5. The DSA lightning indexer was numerically broken for multi-sequence
batches once the top-k mask bites (n_kv > top_k): c4096 n_seq=2 PPL 62.6 vs
dense 2.54, while single-seq was fine. Root cause: the attention-sink
force-include boosted the GLOBAL key range [0, n_sink) by +1e20, which only
protects sequence 0's sink. With several sequences packed contiguously into one
ubatch (seq 0 at cells [0,n0), seq 1 at [n0,n1), ...), every non-first
sequence's sink lives at cell n0.. (not cell 0), got no boost, and was dropped
from top-k once the mask bites — collapsing that sequence (chunk[2]=61.2 while
chunk[1]=2.33).
The cache write and score/argsort were already per-sequence correct: tokens are
placed contiguously like the main K cache, and the base KQ_mask (filled from
kv_self.cells[i].has_seq_id) already drives cross-seq keys to -inf before
argsort. Only the sink was anchored at the wrong (global) cell.
Fix: replace the global arange sink boost with a per-graph input tensor
inp_dsa_sink {n_kv, n_tokens} (F32), filled on the CPU in llama_set_inputs from
kv_self.cells exactly like the KQ_mask:
inp_dsa_sink[j,i] = 1e20 iff cell[i].pos in [0,n_sink) AND
cell[i].has_seq_id(seq_of_query_j), else 0
so each query force-includes only its OWN sequence's sink. For a single
contiguous sequence from pos 0 this is exactly the old "cell index < n_sink"
set with the same magnitude, so n_seq==1 is byte-identical.
Validation (3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1, wikitext-2):
- c4096 n_seq=2 indexer chunk[2]: 61.2 -> 3.07 (== single-seq 3.05).
- c2048 topk=1024 (mask bites): n_seq=4 == n_seq=1 chunk-for-chunk
(2.5005/2.6080/2.7759/3.1137 vs .../3.1138) -> multi-seq is numerically
identical to processing each sequence alone.
- c512 n_seq=1 indexer ON == dense, all 4 chunks byte-identical (no regression).
n_seq=4 at full c4096 (n_kv=16384) OOMs the P100 compute buffer (capacity, not
correctness; n_seq=4 proven correct at c2048/n_kv=8192).
GLM-5.2 DSA indexer is now sequence-correct for n_seq>=1, prefill+decode,
soft_max+FA, -mla 1/-mla 3. Fully general and PR-ready.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: UPDATE 6 — serving-correctness (kr_l maintained across shift/defrag/seq-ops; per-seq sink on first-present pos)
An adversarial review found the indexer was proven on the perplexity path but
not the serving path: the persistent indexer-K cache kr_l was written/read but
never *maintained* by the KV-cache mutators, and the attention sink anchored on
absolute pos<n_sink (wrong after multi-turn seq_rm). This closes those gaps and
pins down what is actually reachable on the MLA model.
kr_l maintenance:
- build_k_shift (llama-build-context.cpp): rotate the indexer keys by the same
per-cell delta as the main K. The cached key is H*concat(RoPE(k_pe,pos),k_nope),
so un-Hadamard (H sym/orthonormal => H*H=I) -> RoPE-delta the pe sub-block ->
re-Hadamard. Exact because GLM-DSA has no rope-scaling metadata (ext_factor=0,
attn_factor=1, freq_scale=1), so NEOX RoPE is pure/composable. Params mirror the
forward indexer RoPE exactly (rope_factors=nullptr); no DEEPSEEK2 yarn-shift leak.
Non-in-place (cont->rope->concat->re-Had->cpy), no aliasing. K-shift Hadamard
input filled in llama_set_k_shift with the identical Sylvester construction.
- build_defrag: kr_l row-move mirrors the k_l move (defrag never changes pos, so
no re-RoPE). max_moves divisor 6->9 *n_layer when the indexer cache is present.
- seq_rm/seq_cp/seq_keep are metadata-only (verified) so kr_l rows stay matched to
cells; seq_add/seq_div set has_shift and route through K-shift. No seq-op change.
Per-seq sink (llama.cpp llama_set_inputs): anchor on each sequence's FIRST PRESENT
pos (min present pos over the scored n_kv span), not absolute pos<n_sink. After
multi-turn seq_rm drops a sequence's early tokens its earliest survivor has
pos>=n_sink; the absolute test would protect nothing. Fresh seq at pos 0 => min=0
=> byte-identical to the old behaviour.
Serving-shift finding (the whole point): a RoPE context-shift on this model is
REFUSED BY THE ENGINE. get_can_shift() returns false for all MLA models
(is_mla_model() includes GLM_DSA); llama_kv_cache_update returns 1 ->
"main : failed to eval". Reproduced AND isolated with a dense control
(DSA_INDEXER_DISABLE=1): dense fails identically at the same token. The failure is
pre-existing MLA engine behaviour, independent of the indexer. On the MLA path the
shift never happens, so the indexer's kr_l can never desync via K-shift; the
build_k_shift kr_l block is correct-and-dormant (documented loudly in code).
Validation (3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1, GGML_CUDA_NO_PINNED=1,
numactl --interleave=all, wikitext-2):
- No regression: c512 n_seq=1 indexer ON == dense == 2.1957 +/- 0.12031,
byte-identical all 4 chunks (2.2770/2.8741/2.3956/2.1957).
- Multi-seq: c4096 n_seq=2 chunk[1]=2.33 chunk[2]=3.07 healthy (== UPDATE 5;
per-seq sink change did not regress).
- Serving shift: engine-refused for MLA, dense control fails identically.
- Independent adversarial review: GO, no correctness defect in the diff.
- Build clean (llama-cli, llama-perplexity, sm_60).
Comments updated (build_deepseek2.cpp): multi-seq+FA no longer limitations; sink
description matches per-seq min-pos anchoring; BIG=1e30 masks on both soft_max and
FA paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: UPDATE 7 — FIX latent graph-reuse cache-fixup omission for the kr_l indexer cache
update_cache_copies() re-points the K/V cache writes to the current kv_head whenever a
compute graph is REUSED (can_reuse_graph reuses iff kv_self.n == prev->n_kv). The persistent
indexer-key cache write (kr_l) is a separate ggml_cpy whose destination view bakes kv_head at
graph-build time, and it was NEVER registered for that fixup. Under FA the cache pads to 256,
so consecutive single-token decode ubatches share the same padded n_kv and the graph IS reused;
without the fixup the kr_l write keeps landing in the first ubatch's slot and later ubatches
never populate their own recent index-key cells (those cells stay at the alloc-zeroed 0.0).
Structurally identical to the MiniMax MSA bug (fork commit 133d14c9).
Fix (mirrors the K/V cache_copies fixup, same shape as MSA 133d14c9):
- llama-context.h: new std::vector<CacheCopy> dsa_cache_copies.
- llama.cpp ctor: resize dsa_cache_copies to n_layer (null entries -> no-op when DSA off).
- build_deepseek2.cpp: register the kr_l ggml_cpy as dsa_cache_copies[il] = {kr_cpy, kr->nb[1]}.
- llama.cpp update_cache_copies(): re-point each registered cpy view_offs = kv_head*step and
patch src[1]->data/data, exactly like K/V, with the c.cpy->view_src == kv_self.kr_l[il]
(+ null/op) guard the MSA fix omitted. soft_max / non-DSA paths byte-identical.
Validation (GLM-5.2-UD-IQ2_M, 3x P100 -ngl 99 --cpu-moe -t 32, NO_PINNED, P2P-disable patch
re-applied to get a working multi-GPU baseline — see UPDATE 7.3; that patch was lost in the
upstream rebase and is required separately):
- c512 -fa1 -mla3 indexer ON: 2.1983 (== prior baseline; build healthy).
- Long-ctx FA decode, 2735-tok recall prompt, -mla3 -fa1 temp0, reuse ON (default): coherent,
correct deep-context recall ("Dr. Mariana Velasquez ... Daniel Okonkwo") on BOTH the fixed and
the unfixed binary.
- ub128 PPL -fa1 -mla3 reuse ON, unfixed: 1.7239/1.8211/2.1888/2.4517, healthy (no inflation).
Honest scope: the bug is real in code but LATENT for GLM-DSA at its configured top_k=2048
(permissive selection keeps the genuinely-attended recent blocks even when reuse leaves some
recent index-key cells stale), unlike MSA's tighter top-k where it inflated PPL ~2x. The fix is
correct and prevents the latent corruption from biting at any tighter top_k / longer ctx /
future serving config. The pre-P2P-patch "nan" seen at ub128 was P2P corruption, not this bug.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-DSA: convert sparse-attention control from env vars to CLI args (off by default)
Implements ikawrakow's direction from discussion #2040: the DSA sparse
indexer must be controllable via command-line argument (not environment
variables), and must be OFF by default for now.
Control surface, before -> after:
DSA_INDEXER_DISABLE (env, inverted: on-by-default) -> --dsa / -dsa
(cparams.dsa, default false; opt-in, dense-by-default)
DSA_TOPK_OVERRIDE (env) -> --dsa-top-k N / -dsatk N
(cparams.dsa_top_k, default -1 == model's configured indexer_top_k)
DSA_HADAMARD_DISABLE, DSA_SINK (env) -> kept as DEBUG-ONLY env
knobs (clearly commented; no CLI surface, not system on/off controls)
Plumbing mirrors existing boolean/int feature flags (-mla, -khad):
include/llama.h llama_context_params {bool dsa; int dsa_top_k;}
src/llama.cpp default_params (false / -1); cparams assignment
src/llama-cparams.h llama_cparams {bool dsa=false; int dsa_top_k=-1;}
common/common.h gpt_params {bool dsa=false; int dsa_top_k=-1;}
common/common.cpp arg parse + help text + cparams copy
src/graphs/build_deepseek2.cpp gate now checks cparams.dsa instead of
getenv; top-k override reads cparams.dsa_top_k. Stays arch-gated to
LLM_ARCH_GLM_DSA. When --dsa is off (default) the indexer function is
never called -> existing dense MLA path, byte-identical to no-feature.
Validation (GLM-5.2-UD-IQ2_M, 3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1,
wikitext-2, 4 chunks @ c2560):
--dsa OFF (default, dense): PPL 2.4151 (graph nodes 4166)
--dsa ON, default top_k=2048: PPL 2.4697 (graph nodes 8846)
--dsa ON, --dsa-top-k 1024: PPL 3.5107
Off-by-default runs the dense path; ON activates the indexer (node count
jumps, PPL shifts as the top-k mask bites once n_kv > top_k). No env var
is consulted for the primary on/off or the top-k knob.
Graph-parallel (-sm graph) interaction (the item ikawrakow flagged):
Under -sm graph the MLA layers are TP-split (wo->extra) and route to
build_deepseek2_tp_attention(), which contains NO indexer code. So --dsa
is silently a NO-OP under -sm graph: it does not error or crash, it runs
dense. Empirically, --dsa --dsa-top-k 1024 under -sm graph gives
PPL 2.4308 (chunks 1.6967/1.7906/2.1664/2.4308) -- the dense baseline
(2.4151), NOT the DSA top_k=1024 numbers (3.5107). The 0.016 delta is
f16 TP-reduce numerics, not DSA. Conclusion: DSA "works under deepseek2"
only on the non-TP (layer) path; serving DSA with -sm graph would require
wiring the indexer into the TP attention path (or a dedicated DSA arch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-DSA: warn that --dsa is inactive under -sm graph/attn (TP path runs dense MLA)
The DSA lightning indexer is built only in the layer-mode (non-TP) attention
path. Under -sm graph / -sm attn the tensor-parallel attention path has no
indexer, so --dsa would silently run dense MLA. Emit a clear one-time
LLAMA_LOG_WARN at context creation instead of degrading silently.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-DSA: drop in-tree dev reference docs from the PR branch
DSA_REFERENCE.md and the R740 progress note are development scratch, not
part of the submission. Remove them so the PR diff is code-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-DSA: fix CPU-only crashes in the sparse-attention path
PR #2045 adds GLM-DSA sparse attention but was validated on CUDA (--cpu-moe).
A CPU-only build (-ngl 0 --dsa) crashes in four spots where the CUDA backend
tolerates something the CPU backend does not. These make GLM-5.2 --dsa run
coherently on CPU; with --dsa off they are no-ops (DSA CPU path only).
1. set_rows into an F32 dest segfaults (ggml.c set_rows_f32):
type_traits[F32].from_float is NULL, so the DSA sparse-mask scatter calls a
NULL fn (segfault at ip=0). memcpy when the dest is F32. CUDA has a real F32
set_rows path, so this only bit the CPU build.
2. ggml_add(F32 score, F16 mask) aborts on CPU (build_deepseek2_dsa_indexer and
build_deepseek2_dsa_sparse_mask): under -fa 1 the dense KQ_mask is F16 and CPU
add only accepts F32+F16 when src0 is F16. Cast the causal mask view to F32.
CUDA's add accepts the mixed types.
3. dsa_fa_mask dim-1 concat must be F32 on CPU (build_deepseek2_dsa_fa_mask):
CPU ggml_concat only supports F16 along dim 0; do the row (dim-1) concat in
F32 then cast the result to F16. CUDA supports the F16 dim-1 concat.
4. indexer k_norm epsilon is 0 -> ggml_norm aborts (llama-hparams.cpp): the
lightning-indexer k_norm is a non-RMS LayerNorm using f_norm_eps, but the
GLM-DSA GGUF only carries the RMS eps so f_norm_eps stays 0
(GGML_ASSERT(eps > 0)). Mirror the RMS eps. CUDA's norm doesn't assert on eps=0.
Validated: GLM-5.2 UD-Q4_K_M, single-socket Xeon w7-2475X, CPU-only (-ngl 0 --dsa)
- coherent at 49K+ ctx, correct 30K needle retrieval, prefill flat with length
(~32 tok/s, the O(L) DSA signature) vs the dense build's O(L^2) decline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* DSA: loop over attention heads + use builtin Hadamard
* DSA: ggml_blend
* DSA: remove a bunch of unnecessary ggml_cont
* DSA: fix CUDA blend - but something is still wrong
* DSA: use ggml_top_k instead of ggml_argsort when FA is ON
* CUDA: add CUB based argsort
* DSA: avoid graph leaves
* Various
* GLM-5.2 DSA: IndexShare (shared layers reuse full-layer top-k)
GLM-5.2's indexer_types marks 21 'full' layers that compute their own
lightning-indexer top-k and 57 'shared' layers that reuse the previous
full layer's top-k. This port computed an independent top-k on every
layer, which mis-selects keys on the 57 shared layers (the transformers
reference sets indexer=None on shared layers and reuses prev_topk).
Shared layers now reuse the most-recent full layer's selection. Full/
shared map derived from the config rule (full iff il<=1 or il%4==2),
which reproduces indexer_types exactly; loader can later override from
GGUF metadata. Built on #2063's tree; head-loop/ggml_hadamard/ggml_blend/
argsort/FA-mask unchanged.
4K PPL (unsloth IQ2_M, top_k 2048, CPU): DSA-on 3.1922 -> 2.7111, dense
2.6972 (~97% of the gap). top_k>=n_kv reproduces dense exactly. Single-
seq and 4x8 parallel decode coherent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply suggestion from @ikawrakow
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: mgkwill <168222+mgkwill@users.noreply.github.com>
Co-authored-by: Kawrakow <iwankawrakow@gmail.com>
The CPU `set_rows` kernel for F32 sources fetches `type_traits[dst->type].from_float`
and calls it for every scattered row. F32 has no `from_float` entry, it is NULL in
`type_traits`, so any `set_rows` into an F32 destination calls a NULL function pointer
and segfaults. Other destination types work because they all have a real `from_float`.
Repro (CPU backend, standalone ggml graph):
dst = new_tensor_2d(F32, 8, 6) // F32 destination
src = new_tensor_2d(F32, 8, 4)
idx = new_tensor_1d(I64, 4) // {0,2,4,5}
out = ggml_set_rows(dst, src, idx)
// ggml_backend_graph_compute(cpu, ...) -> SIGSEGV on current main
When the destination is F32, copy the row with `memcpy` instead of going through
`from_float`. The I32 and I64 index branches both get the same treatment. An assert
guards the remaining case, non-F32 dst with a NULL `from_float`, so a future
unsupported type fails loudly instead of crashing.
I ran a normal model after this and it still decodes fine (DeepSeek-V2-Lite-Q4_K_M,
CPU, coherent output), and the non-F32 path is untouched. On the F32 path you pay one
`memcpy` per row in place of the indirect call.
Co-authored-by: local-llm <local-llm@local-llm-R740.cruvis.org>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The flash-attn vec kernels walk the KV cache in blocks of Dk rows for the
score loop but accumulate V in blocks of Dv. With Dk == Dv that is the same
thing, so normal attention shapes are fine. For absorbed MLA shapes where the
K and V head sizes differ (Dk=576/Dv=512 and Dk=192/Dv=128) the two loops step
a different number of KV rows, so K and V drift out of sync after the first
block and the V pointer reads the wrong cache rows.
This only shows up at decode (batch=1) on cards that fall back to the vec
kernel for MLA, which on NVIDIA is pre-Volta. There deepseek2/GLM MLA models
with -mla 1 -fa 1 or -mla 3 -fa 1 decode coherently for short prompts but
collapse into garbage once n_kv passes the first KV block (Dk=576). Prefill/PPL
is unaffected because prefill takes the tile kernel, not the vec kernel.
Fix: the score loop already covers Dk KV rows, so the V loop and the V pointer
step Dk rows too. For asymmetric Dk>Dv the V row is only Dv wide, so threads
with tid >= Dv have no V element (their VKQ lane is discarded at the output
store anyway) and read 0 instead of stepping past the row.
The change keys off the compile-time Dk != Dv, so every symmetric instantiation
compiles to byte-identical code and modern GPUs (which never take this vec path
for MLA) are unaffected.
Validated on a Tesla P100 (sm_60) with DeepSeek-V2-Lite Q4_K_M: decode coherence
restored for -mla 1/3 -fa 1, KLD vs the -fa 0 soft_max path drops from 4.79 to
1.4e-4 (same top token 27% -> 100%) at c1024, and TG is unchanged (82.8 t/s).
Co-authored-by: mb8565 <244351746+mb8565@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* WIP: Split mode graph for Gemma4 assistant
Something is not right - acceptance drops to nearly zero.
* Per model CUDA contexts
Still not working!?
* This works
The issue was that I was not correctly calculating the number
of KV heads for the split KV cache.
* Compiler warnings
* It is better to use llama_context pointers as keys
* fix: wrong stride in batched quantized add1 (nb0 -> nb3)
ggml_compute_forward_add1_q_f32 used i3*nb0 (element stride) instead of
i3*nb3 (batch stride) for the destination row pointer. This causes all
add1 operations with quantized types and batch > 1 to write to wrong
memory locations. The src0 pointer on the line above correctly uses nb03.
* fix: wrong dimension limits in dup_f16 non-contiguous path
The destination index wrapping in ggml_compute_forward_dup_f16 used
source dimensions (ne00/ne01/ne02/ne03) instead of destination dimensions
(ne0/ne1/ne2/ne3). While source and destination shapes are currently
identical for dup, using the wrong variables is incorrect by design.
* fix: wrong dimension limits in dup_bf16 non-contiguous path
Same fix as the dup_f16 path: destination index wrapping used source
dimensions (ne00/ne01/ne02/ne03) instead of destination dimensions
(ne0/ne1/ne2/ne3). Copy-paste error from the contiguous path.
* fix: ACC work size uses src[1] instead of src[0]
The dequantization work buffer for quantized ACC was sized using
src[1]->ne[0] instead of src[0]->ne[0]. Since src[0] is the tensor
being dequantized, its dimensions should determine the buffer size.
* fix: missing work size for SOFT_CAP_MAX and ROPE_BACK
Both ops dereference params->wdata in their forward functions but had
no work size allocation (cur = 0), causing a NULL pointer dereference
when any thread attempted to use wdata.
* fix: wrong dim in sum_rows_f32 dimension decomposition
Line 14404 used ne01*ne0 (= ne01*1) instead of ne01*ne02 for the
i3 term in the flat row index formula. When ne02 > 1 (batched 2D
inputs), this causes wrong memory access and corrupted results.
* fix: wrong tensor index in BF16 fused RMS norm add path (norm.cu:1039)
The BF16 branch of ggml_cuda_op_fused_rms_rms_add used dst->src[2]->data
for the second weight pointer, but should have used dst->src[3]->data.
This caused reading float weights from the wrong bf16 input tensor.
The F32 and F16 branches both correctly reference src[3], and the
assertions at lines 1013-1015 confirm src[3] is the F32 weight tensor.
* fix: off-by-one bounds check in 7 dmmv kernels (row > nrows -> row >= nrows)
Seven K-quant dequantize_mul_mat_vec kernels used row > nrows for bounds
checking instead of row >= nrows. Since rows are 0-indexed (0..nrows-1),
the check missed the row == nrows case, allowing a potential out-of-bounds
memory write when grid dimensions produce exactly nrows.
The templated dequantize_mul_mat_vec<type> kernel at line 667 already used
the correct row >= nrows pattern.
* fix: typo in function name iqk_mul_mat_vec_q_kerne -> iqk_mul_mat_vec_q_kernel
Truncated function name in iqk_mmvq_templates.cuh was missing trailing 'l'.
* fix: print actual split_dim value in set_tensor error message (ggml-cuda.cu)
fprintf used extra->split_dim == 0 which evaluates to boolean 0 or 1
instead of the actual split dimension value. When this fatal error is
hit for an unsupported split_dim, the user could not diagnose which
value caused the problem.
* fix: wrong src index in gate bias stride for fused up-gate MoE path
ggml_cuda_add_id for the gate bias used dst->src[4]->nb[1] as the stride
argument instead of dst->src[5]->nb[1]. This was a copy-paste error from
the up-bias code (lines 3220-3224) where src[4] is correct. If src[4]
and src[5] have different strides, the bias addition produces incorrect
results.
* fix: wrong row count for gate projection MMQ in fused up-gate MoE path
ggml_cuda_op_mul_mat_q for the gate projection (src0_2) used
src0_1->ne[1] as row_high instead of src0_2->ne[1]. This copy-paste
error causes processing the wrong number of rows if the up and gate
projections have different row counts. The gemv path (line ~3563)
correctly used src0_2->ne[1].
* CUDA : typo
* CUDA: Add missing GGML_CALL to function definition
* CUDA: only log GGML_CUDA_FORCE_MMQ/CUBLAS when enabled
* CUDA: Fix softcap bug in flash_attn_tile_ext_f16
The else branch (softcap != 0) incorrectly called launch_fattn_tile_f16_64_128
with use_softcap=false instead of true, causing logit softcap to be silently
ignored for the col_per_block=32, parallel_blocks=1 path.
* host-swap tensor loop
the host-swap functionality is only triggered when the certain env. variables are declared
* target_include_directories tweak
* hot-swap tensor support
two intrusions:
1.) at the model loading to collect the snapshot
2.) the modification of the `/health` HTTP endpoint to be able to trigger the hot-swap via sending the `llama-server` the HTTP-request.
*both a braced by the specific env. variables
* hot-swap tensor support; graph invalidation
ggml_backend_cuda_invalidate_graphs export
* hot-swap tensor support
graph invalidation implementation; extended debug output (commented out)
* llama_reload_changed_tensors export
* tensor hot-swap on-demand reload
cpu-only/hybrid/gpu-only with split mode layer/graph full support implementation
* docs
* reuse the gguf parsing from llama.cpp
gguf_init_from_file, gguf_find_tensor, ggml_get_tensor
* remove the manual scheduling for hybrid inference
* update docs
* tensor shape validation
* update docs
* update docs
accidentally wiped the previous changes; so recovered them
* revert the GGML_CUDA_MAX_DEVICES to 16
* update llama_reload_changed_tensor
update llama_reload_changed_tensor, revert CMakeLists.txt
* update llama_reload_changed_tensor
* GGML_MAX_SRC
GGML_MAX_SRC compile-time definition support
* GGML_MAX_SRC
GGML_MAX_SRC compile-time definition support
* GGML_MAX_SRC
GGML_MAX_SRC compile-time definition support
* llama_reload_changed_tensor
update llama_reload_changed_tensor definition
* refactory
move the tensor-reloading implementation to llama-reload.cpp, llama-reload-info.h; some bugfixes and code reduction
* revert
added back the missing newline
* update docs
* reload_info constructor
* bugfix: cpu-only
TODO: improve the working environment by compiling for multiple hardware configurations; possibly make a test pipeline
* cpu-only bugfix
set the fix again after unsuccessful sync with main
* windows os compilation fix
#include <string>
* fix windows os build
error C2039: 'string': is not a member of 'std'
* remove dead file
* implement perplexity in server
* Revert "implement perplexity in server"
* AVX VNNI auto-activation
Enables auto-detect of AVX VNNI and its definition in the CMakeLists
Detected by ik_llama.cpp.
* IQ4_XS R8: Enable AVX-VNNI 256-bit path with MSVC compatibility
Migrate mul_mat_iq4_xs_r8_q8_k_avx2() from HAVE_FANCY_SIMD to HAVE_VNNI256.
Changes (6 guard sites + 8 intrinsic calls in iqk_gemm_kquants.cpp):
- Replaced 3x #ifdef HAVE_FANCY_SIMD with #ifdef HAVE_VNNI256
- Replaced 3x #ifndef HAVE_FANCY_SIMD with #ifndef HAVE_VNNI256
- Replaced 8x raw _mm256_dpbusd_epi32 with ggml_mm256_dpbusd_epi32
(the ggml wrapper resolves to _mm256_dpbusd_avx_epi32 on MSVC via
the iqk_config.h macro, which is the correct MSVC AVX-VNNI intrinsic
available under /arch:AVX2; raw _mm256_dpbusd_epi32 does not exist
in MSVC headers without AVX-512)
Impact:
- IQ4_XS_R8 matmul now uses VNNI256 on CPUs with AVX-VNNI but no
AVX-512 (e.g. Intel Arrow Lake / Core Ultra 265K)
- Previously limited to HAVE_FANCY_SIMD (full AVX-512) exclusively
- This path is exercised when models are loaded with -rtr / --run-time-repack
(in-memory repack) or when using --repack to create a permanent IQ4_XS_R8 file.
Standard IQ4_XS does not auto-convert to IQ4_XS_R8 at load time.
* Qx_0 R4 legacy quants: Enable VNNI256 path for AVX-VNNI CPUs with MSVC compatibility
Three changes in iqk_gemm_legacy_quants.cpp:
1. DotHelper (line 23): Extend VNNI condition to include HAVE_VNNI256
(not just __AVX512VNNI__+VL) and use ggml_mm256_dpbusd_epi32
wrapper for MSVC compatibility. This fixes Q6_0 non-R4 path
and all other quant types routed through UnsignedDot/SignedDot.
2. accum_q4_0_quants (line 994), mul_mat_q5_0_r4_q8_2_avx2
(lines 1202, 1223), mul_mat_q6_0_r4_q8_2_avx2 (lines 1375, 1394):
Replace #ifdef HAVE_FANCY_SIMD / #ifndef HAVE_FANCY_SIMD with
HAVE_VNNI256 (which correctly detects AVX-VNNI without requiring
full AVX-512). Also replace raw _mm256_dpbusd_epi32 with
ggml_mm256_dpbusd_epi32 wrapper.
These paths were dead code on Arrow Lake (HAVE_FANCY_SIMD requires
full AVX-512 which Arrow Lake lacks). Now they compile and use
the hardware VNNI instruction (vpdpbusd) via __AVXVNNI__.
Note: remaining HAVE_FANCY_SIMD guards in this file guard true
AVX-512 paths (_mm512_* intrinsics) and are left unchanged.
* Simplify def
Analogous to the BF16 fix in eea6a82b25, this adds proper Q8_0
type handling in ggml_cuda_op_add:
- Add k_add_q8_0_f32 kernel: dequantize Q8_0, add F32, store F32
- Add k_add_q8_0_q8_0_f32 kernel: dequantize two Q8_0, add, store F32
- Add Q8_0+Q8_0/Q8_0+F32/F32+Q8_0 branches in the F32 dst (else) block,
preventing Q8_0 data from falling through to the incorrect half cast
- Expand Q8_0 dst branch to handle F32+Q8_0->Q8_0 (swapped args), not
just Q8_0+F32->Q8_0
The MMA flash-attention dispatcher only instantiated ncols2 = 8 and 4 for
head_dim 512, so any other GQA ratio hit GGML_ABORT. Gemma 4 12B's global
attention layers use head_dim 512 with a 16:1 GQA ratio (16 query heads /
1 KV head), which aborts at load. Because MTP speculative decoding requires
flash attention, this also blocks the Gemma 4 12B MTP drafter entirely.
Instantiating ncols2 = 16 there is not viable: it exceeds the maximum dynamic
shared memory on Ada (cudaFuncSetAttribute returns invalid argument). Instead,
route gqa_ratio % 8 == 0 (covering 8 and 16) through the existing ncols2 = 8
kernel, which already iterates over Q-head groups (iter_z = ceil(gqa_ratio /
ncols2)). gqa_ratio 8 and 4 behavior is unchanged; this mirrors the divisor
dispatch already used for the 576x512 case below.
Verified on RTX 4070 Ti SUPER (Ada, cc 8.9): Gemma 4 12B + MTP drafter now
runs with flash attention; draft acceptance 43-95% by workload, 1.5-2.2x
end-to-end speedup. The 26B-A4B drafter (gqa_ratio 8) is unaffected.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Disable K Hadamard transform if K-head size is not a power of 2
* Allow Hadamard transform for head sizes that are not power of 2
* Give more details why Hadamard is not possible
* Arghh
* fa: fix FlashQKV early-termination causing S=0 assertion with --parallel N>1
The backward-scan optimization in compute_helper/compute_helper_q checks
only one mask position per k_step block on the last query row (q_step-1)
to find where valid KV entries end. When q_step > 1 and different query
rows have non-overlapping valid KV regions (multi-slot / --parallel N>1),
the scan on the last row's mask can miss blocks that contain valid entries
for earlier rows. This causes those rows to accumulate S=0, triggering
the GGML_ASSERT(S > 0) in normalize_and_store_1row.
Fix: remove the early-termination scan at all 4 sites and iterate all
nk1/k_step blocks unconditionally. The mask already handles correctness:
fully-masked blocks produce smax=-inf and skip V accumulation, so the
performance cost is minimal for TG (small nq1) and acceptable for PP.
Fixes#809
* fa: refactor multi-slot mask fix into mask_effective_nk1() helper
Replace 4× inlined early-termination scans with a shared helper that
computes the effective K boundary by scanning ALL query mask rows
(union-of-masks). This is the minimal fix for multi-slot parallel
inference where different slots have different sequence lengths.
The helper returns the k_step-aligned boundary covering the longest
active sequence across all rows, preserving single-slot performance
(single row = same boundary as before).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Turbomen008 <Turbomen008@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* ggml: ggml_dequant_hadamard fused op for MLA -khad path
Adds a new ggml op that fuses (ggml_cast -> F32) + (ggml_hadamard) into a
single kernel. Reads a quantized (or F16/F32) source and produces a per-
Hadamard-block F32 chunk with the inverse transform applied, without
materializing a full-size F32 intermediate buffer.
Motivation: the MLA pp_opt path in build_deepseek2.cpp un-encodes the
H-applied cache_nope view at every PP call. Today that runs as a cast
(quant -> F32) followed by a separate ggml_hadamard kernel, costing two
full-size F32 passes per layer per rank per call. Fusing them halves
the bandwidth on the un-encode and removes one kernel launch.
CUDA kernels in dequant_hadamard.cu lift the Walsh-Hadamard butterfly
from hadamard.cu and dequant helpers from dequantize.cuh:
* qr=1 layout (q8_0): consecutive dequant pair, stage 1 fused with load
* qr=2 layout (q4_0 / q4_1 / q5_0 / q5_1 / q6_0 / iq4_nl): dequant pair
at stride qk/2, explicit stage 1 after sync
* F16 has a dedicated kernel
* F32 source falls back to the standalone Hadamard op
CPU impl in iqk_cpu_ops.cpp composes the existing type_traits.to_float
dequant with fast_ht for graph completeness. nh in {64, 128, 256, 512}.
* MLA-TP: Hadamard pretransform of wv_b/wk_b_pp for -khad
Fold the 64-block orthonormal Hadamard into wv_b and wk_b_pp once at
context init so the pp_opt mul_mats consume the K cache in its on-disk
encoded basis. The per-PP-call cache_nope un-Hadamard is then skipped
(rope half still un-applied — it goes to FA via concat, no wk_b multiply).
Math is identity by H^T H = I: mul_mat(H@wv_b, H@cache) = wv_b^T @ cache.
For mla=2/3 absorb, composes correctly with the existing post-FA
ggml_hadamard(kqv_compressed, 64).
All-or-nothing across layers under a castable type-allowlist (excludes
1-3 bpw IQ types whose requant blows up beyond PPL noise). Models with
ineligible weights fall back to the runtime un-Hadamard path unchanged.
Composes with the fused ggml_dequant_hadamard op (prior commit): with the
fold active only the rope half still runs the runtime transform, via the
fused kernel.
* MLA-TP: fix TG with -khad after wv_b/wk_b_pp fold
The absorb branch of build_deepseek2_tp_attention applies
ggml_hadamard to kqv_compressed after FA, then multiplies by
wv_b. Pre-fold this was needed because wv_b was un-encoded; with
the wv_b fold (prior commit) the mul_mat already expects
H-encoded kqv_compressed:
mul_mat(H @ wv_b, kqv_encoded) = wv_b^T @ H @ H @ kqv_unencoded
= wv_b^T @ kqv_unencoded (H @ H = I)
Skip the post-FA hadamard when model.khad_pretransformed is set
so the two H applications cancel instead of double-applying.
Affects the absorb branch: TG (n_tokens=1), short-context PP
(n_kv < 1024), and models without wk_b_pp. Long-context PP goes
through the pp_opt branch and is unrelated/unchanged.
Reported by @ikawrakow on PR 1852. Verified across mla={1,2,3} x
khad={on,off} x -ctk={q8_0,q4_0} on GLM-4.7-Flash IQ5_K and the
unsloth IQ4_XS variant ik used to reproduce.
* ggml_hadamard: accept F16 and quant sources; drop GGML_OP_DEQUANT_HADAMARD
Per @ikawrakow review on PR 1852: subsume the per-source-type dispatch
into the existing GGML_OP_HADAMARD instead of carrying a separate enum
entry, op constructor, and standalone files.
ggml_hadamard's API is unchanged from the call-site perspective. The
constructor's F32-only assertion is dropped; ggml_cuda_op_hadamard and
iqk_hadamard now dispatch internally:
- F32 source: existing F32 butterfly (unchanged)
- F16 source: dedicated kernel
- q8_0 / q4_0 / q4_1 / q5_0 / q5_1 / q6_0 / iq4_nl: fused dequant +
butterfly kernel (lifted from the deleted dequant_hadamard.cu)
- CPU side composes traits.to_float with fast_ht
Net diff: -80 lines. Removes dequant_hadamard.{cu,cuh}, the enum entry,
op table rows, ggml_dequant_hadamard constructor, dispatch cases, and
the DEQUANT_HADAMARD supports_op block.
Verified clean build + TG smoke (mla=3 +khad q8 on GLM-4.7-Flash-IQ4_XS,
same coherent output as prior commit on feat/dequant-hadamard).
* MLA tensor parallelism under -sm graph (DEEPSEEK2/GLM_DSA/MISTRAL4)
Extends -sm graph (split-mode graph) to MLA-style attention across the
DEEPSEEK2, GLM_DSA, and MISTRAL4 architectures. Previously these archs
fell back to -sm layer regardless of the user's flag.
Implementation:
- Per-rank attention build in build_deepseek2_tp_attention with
view-sliced FlashAttention, split-buffer output projection, and
ggml_reduce across devices
- wk_b / wv_b absorbed weights replicated per device via materialize()
in llm_prepare_mla (these can't live in a split buffer)
- KV cache replication path (replicated_k_l) for graph-mode TP
- distribute_mla_tensors_for_split_mode_graph routes attention/norm
tensors into ctx_split; expert tensors stay per-layer
- Implements ggml_backend_cuda_split_buffer_get_tensor for the
replicated / row-split / col-split inverse paths
- Early-reject guard in src/llama.cpp that auto-downgrades -sm graph
to -sm layer (with a warning) when incompatible loader flags are set:
-ncmoe, -cmoe, -ot, -rtr, -muge
New CLI flag:
- -gap | --graph-attn-precision <f16|f32> (default f16)
See the PR description for the full validation matrix (3 archs x 2/4/8
GPU counts), perf numbers, VRAM accounting, and known limitations.
* Some tweaks
* materialize lambda: per-head split for graph-mode tp_replicate
7dd19e19 changed wk_b/wv_b distribution from mirror to per-head split
(split_dim=2) via prepare_split_tensors. That path only fires when
wk_b/wv_b are loaded from GGUF.
Models that store only wkv_b in GGUF derive wk_b/wv_b at load via
llm_prepare_mla, going through the materialize lambda, which was
untouched and still produced mirror replicas (split_dim=-1, full n_head
per device).
build_deepseek2_tp_attention now does mul_mat(wk_b_local, q_nope_perm)
without the prior view_3d slice, so a mirror replica passes an n_head
tensor where the kernel expects n_head_local. Result: silent SIGSEGV
right after model load.
Mirror logic in materialize is replaced with the same per-head split as
prepare_split_tensors: head_offsets derived from wo split, each rank
gets a tensor with ne[2]=n_head_local, data copied from the appropriate
source byte slice. Singular `computed` tensor keeps full metadata for
tensors_by_name lookups.
Tested: 8x3090, -sm graph -mla 3 -fa on now boots cleanly and
sweep-benches without crash. Log confirms new path: "Computed
blk.X.attn_k_b.weight ... split across N devices on dim=2".
* cleanup: indent fix + remove dead view_3d slicing and debug printf
- build_deepseek2.cpp: re-indent the self_attention block in
build_deepseek2_layer_attention (lines 253-670). Block was at column 0
inside a function body; now at the expected 4/8-space indent.
- build_deepseek2.cpp: drop the commented-out view_3d slicing and debug
printfs left over after 7dd19e19's switch to direct mul_mat on
per-rank wk_b_local / wv_b_local. Update the stale 'wk_b is
replicated (split_dim=-1)' comment to match the new split_dim=2
reality.
- ggml-cuda.cu: remove the leftover debug printf in
ggml_backend_cuda_split_buffer_get_tensor.
No behavior change. Verified with a clean rebuild and DSV2.5 +
GLM-4.7-Flash sweep-bench runs.
* llm_load_tensors: gate incompatible-flag warning to MLA archs
The -ncmoe / -rtr / -muge / -ot warning under -sm graph currently fires
for all archs that support graph mode. That's an over-reach: the
incompatibility is specific to the MLA TP paths (DEEPSEEK2, GLM_DSA,
MISTRAL4) — Gemma4 graph mode existed pre-PR and works with those flags.
Gate the warning to MLA archs only.
Also refreshes two stale comments left over from the wk_b/wv_b
mirror -> per-head-split rewrite:
- src/llama.cpp llm_prepare_mla: "Replicate wk_b/wv_b ..." now reads
"Per-head split wk_b/wv_b ..." to match what the materialize lambda
actually does post-823a39e2.
- src/llama-load-tensors.cpp distribute_mla_tensors_for_split_mode_graph:
drop the wkv_b row-split mention (wkv_b is no longer created under
graph mode after 7dd19e19) and correct the wk_b/wv_b distribution
description (per-head split, not per-device replicated).
---------
Co-authored-by: Kawrakow <iwankawrakow@gmail.com>
* Avoid copying the per-step SSM state (CUDA)
* Avoid copying the per-step SSM state (CPU)
* Allocate only what is necessary for per-step SSM state
* Cleanup
* Use AVX version VNNI intrinsic when AVX512VNNI not available.
* remove changes under HAVE_FANCY_SIMD
---------
Co-authored-by: XZiar <xziar@xziar.xziar>
The default of 0x602 (Windows 8) causes a build failure on any toolchain
where _WIN32_WINNT propagates into vendored cpp-httplib (notably MinGW with
the bundled w64devkit GCC). cpp-httplib's httplib.h has, for some time
now, contained:
#ifdef _WIN32
#if defined(_WIN32_WINNT) && _WIN32_WINNT < 0x0A00
#error "cpp-httplib doesn't support Windows 8 or lower. Please use
Windows 10 or later."
#endif
#endif
so the entire llama-server target fails to compile on Windows + MinGW
unless the user passes -DGGML_WIN_VER=0x0A00 manually.
Bumping the default to 0x0A00 (Windows 10) keeps Windows 8 reachable for
anyone who explicitly requests it (-DGGML_WIN_VER=0x602) while letting the
default Windows + MinGW build succeed end-to-end. Windows 8 / 8.1 reached
end of support in January 2023, and Windows 10 is a strict superset of the
Win8 surface used elsewhere (PrefetchVirtualMemory etc.), so this is
strictly additive on the API side.
Verified by building with w64devkit 2.8.0 (gcc 16.1.0) on Windows 11
without any -DGGML_WIN_VER override: all 266 ninja targets link cleanly,
including bin/llama-server.exe, and llama-cli runs Qwen3-4B-Thinking-2507
IQ4_XS at ~6.2 tok/s with q8_0 KV at 4096 context.
* server: spec checkpoints for recurrent models
* fix: save/restore sampler state during speculative checkpoint
When speculative decoding rejects draft tokens and restores the
recurrent state checkpoint, the sampler (RNG, grammar, prev tokens)
must also be restored to maintain consistency. Without this, the
sampler state reflects the rejected draft tokens, leading to
potential divergence.
Uses common_sampler_clone() to snapshot the sampler before the
speculative batch decode, and restores it on rejection.
* server: snapshot recurrent state in tensor
* reset ngram mod state for rejected tokens
* server: refactor checkpoint state logic
* speculative: fix sampler for checkpoints
* recurrent model: implement recurrent kernel checkpoint
* recurrent model: refactor api
* spec: free rbudget before overwriting
The token loop reads sK[] in the state update (bottom of loop) but has
no barrier before the next iteration overwrites sK[] (top of loop).
Without an explicit memory fence, hardware/compiler reordering can
cause non-deterministic reads from shared memory.
Per review: with this barrier in place, the prior __syncthreads() after
the cross-warp reduction and the one immediately after loop exit are
both redundant. The new barrier is a full block-level fence that also
orders all_sum1/all_sum2 reads vs. the next iteration's writes, and
every thread reaches it before leaving the loop. Both redundant
barriers removed.
No performance impact — GPU utilization is 31-33% during inference,
bottlenecked by CPU MoE expert computation, not the CUDA kernel.
Co-authored-by: Mark Alonzo <mark.alonzo@outlook.com>
* WIP: Gemma4 vision
Crashes on the GPU because of rms_norm requiring ne0 to be multiple
of warp_size.
Runs on the CPU, but produces garbage.
* Remove unnecessary assert in CUDA rms_norm
* GLU was not advertised as supported on CUDA
* Still not working
* This seems to work
Kawrakow
Joel Farthing
hchengit
mb8565
cora4
Jun Yamog
Nexesenex
dmaivel
Lorenzo Vasile
Farmadupe
mb8565
magikRUKKOLA
a1batross
Chip Bradford
Gearstickle
David Young
Forkoz
Samuel Oliveira Alves
XZiar
Alex
markaalonzo