* ggml : assert the delta-net value head dim equals the key head dim
The CPU forward sizes its result from the value head dim (src[2]->ne[0]) but
indexes it with the key head dim (src[0]->ne[0]). A model where the two differ
was mis-indexed silently. The CUDA op has asserted this all along.
* ggml : fuse the delta-net recurrent state copy into the op
The problem: the delta-net op produces the new recurrent state into the tail of
its result. Then llama copies that tail into the KV slot the state was read
from. The copy buys nothing - the kernel could write the slot itself.
The change: the slot's two halves are written by two narrow CPY nodes instead of
one CONCAT. That isolates the state write in an ordinary node, which the
scheduler places by the rules it already has. A backend that recognises the
pattern lets the kernel write the slot directly and skips that node. One that
does not implement the fusion runs the copy as before. No public header changes.
Notes: this leaves ggml_concat_inplace(), added in #1777 for exactly this site,
without a caller.
* DS4 optimizations (part 2)
* This is slightly better
* Another minor tweak
* Increase max. number of graph splitinputs to 64
Else with DS4 we can trun into an assert for specific offload
situations with more than one GPU.
* Adding ds4_comp op with CPU implementation
* ds4_comp on CUDA
* ds4_comp: ratio = 4 specialization
Surprisingly small performance gain
* Also handle HCA via ds4_comp
But much smaller gain, if any.
* Delete commented out stuff
* Remove the [(size_t) il] noise
* Minor
* Fix quantized cache
Adds ggml_latent_attn_prefix_ext / ggml_latent_attn_indexed_ext: MLA
latent-cache attention with an always-visible learned K/V prefix
(openPangu's 128 param_sink rows), joint softmax over [prefix | cache],
reading the raw F32/F16/Q8_0 latent cache directly. CUDA implementation
plus a scalar CPU reference that pins the op's semantics; the CPU
backend reports support truthfully, and openPangu adopts the op only on
a non-CPU backend as builder policy.
openPangu routes its dense/SWA/MTP full-span attention and the gathered
DSA path through the op, capability-gated per layer on the attention
output projection's scheduled backend, with the latent cache required
resident on that same backend (--no-kv-offload keeps the unfused
chain); any layer whose backend cannot run the candidate keeps the
exact unfused chain.
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
* indexer_topk: fix quantized q8_1 scratch sizing on CUDA
The quantized-K path under-sized its q8_1 scratch buffer: it allocated q->ne[1]*max_rows blocks using the unpadded head dim, but quantize_mmq_q8_1_cuda writes q_padded/QK8_1 blocks per row and must process all q->ne[1]*nrows rows. Size the buffer by (q_padded/QK8_1) blocks x (q->ne[1]*max_rows) rows and pass the full q->ne[1]*nrows row count so the scratch cannot be overrun and every query row is quantized.
CUDA graph identity: INDEXER_TOPK dispatches a source-type-specialized kernel (dense F16 vs quantized cache, F32 vs F16 mask). Source addresses alone do not identify the captured kernel, so snapshot each source type in ggml_graph_node_properties and force re-capture when an INDEXER_TOPK source type changes, preventing a reused graph from replaying the wrong kernel variant when sources are reallocated at the same address.
CPU backend: report INDEXER_TOPK support via iqk_indexer_topk_supported (guarded by GGML_USE_IQK_MULMAT) so the scheduler places the node on a backend that can run it. Harden the scheduler's pass-5 node-assignment check from assert to GGML_ASSERT so a node no backend supports fails as a defined abort under NDEBUG instead of indexing sched->backends[-1].
* indexer_topk: drop CPU capability predicate
* indexer_topk: fall back when unsupported
* cuda: drop speculative INDEXER_TOPK graph type matching
---------
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
* initial map to load deepseek 4 arch
* wip
* wip: match graph build and attn logic for dpv4
* wip: Enhance DeepSeek-V4 architecture with new tensor types and sqrtsoftplus gating function
* Update DeepSeek-V4 to support raw key indexing with read/write indices
* fix mismatch in attn_raw
* Enable FA with CSA/HCA
* Fix logit mismatch with FA path
* Clean traces and logs for debug
* Refactor DSV4 tensor handling for MTP execution and improve raw context management
* Refactor DeepSeek4 tensor operations: replace manual weighted sum and post-processing with new helper functions
* Share mHC pre-projection and fix packed DSV4 writes
* DSV4: add shared top-k selection and improve mask handling
* Fix DSV4 c2048 view stride and duplicate loader instantiation
* Reuse shared RMS normalization in DSV4 graph
* Replace DSV4 indexer rotation with shared Hadamard
* Share CSA visibility mask with DSV4 LID
* dsv4: document dependency ordering and reset state
* Remove DSV4 zero-dependency graph shim
* Fix DSV4 packed stream execution
* Remove DSV4 l_out backend override
* Enable DSV4 quantized K-only cache
* Revert "Enable DSV4 quantized K-only cache"
This reverts commit 04f9b425321f62ba60e16d1bea2f8de714cfe855.
* Fix DSV4 quantized cache accounting
* Fail closed on unsupported DSV4 cache lifecycle operations
* Various optimizations
* llama: fix GGML_METAL=ON build - missing ggml-metal.h include in llama-dflash.cpp (#2134)
llama-dflash.cpp calls ggml_backend_is_metal() and
ggml_backend_metal_set_n_cb() inside an #ifdef GGML_USE_METAL block but
never includes ggml-metal.h, so any Metal-enabled build fails to
compile. Add the same guarded include llama.cpp already uses.
* New op: ggml_sum_rows_ext (#2132)
* Add ggml_sum_rows_ext
* openPangu: use ggml_sum_rows_ext also in mhc_post
* openPangu: use ggml_sum_rows_ext also in mhc_tail
* Minor
* Reuse shared inverse RoPE operation for DSV4
* Reuse maintainer CUDA concat implementation
* WIP
* hc_pre
* hc_post
* Remove unnecessary mask manipulations
* WIP
* Take into account swiglu limits
* Turn on fused indexer by default
* Give names to mat mul results
* More named ops
* dsv4: do not uselessly copy the KV cache
+20% TG at 32k tokens
* mask_to_index and make CPU FA work with that
* Much better CPU-only, CUDA still not functional
* Better CPU TG
I'm now at 9.7 t/s for zero context and 6.5 t/s for context of 32k.
PP is 120 t/s for short context and 101 t/s at 32k.
* Even better CPU TG
I'm now at 8.1 t/s for context of 32k tokens.
* Turn off DSA on CUDA for now
* Fix CUDA DSA
* Remove again the unnecessary softmax result buffer
* Experiments
* Various
* More named ops
* Forgot to uncomment
---------
Co-authored-by: samuel <samueloliveira32df@gmail.com>
Co-authored-by: hchengit <95317477+hchengit@users.noreply.github.com>
The Metal backend lacked GGML_ROPE_TYPE_MROPE/IMROPE support, so models
whose GGUF carries rope_sections (e.g. Qwen 3.5 hybrids) could not run
fully offloaded on Apple Silicon.
Add rope_multi_f32/f16 kernels with section-based position handling
(t/h/w/e blocks, 4 position ids per token), imrope's interleaved section
selection, and the corresponding dispatch in ggml-metal.m. Vision-mode
mrope is not implemented and is asserted out explicitly.
Validated on M2: kernel output matches the CPU backend, and Qwen 3.5-9B
(Q4_K_M, -ngl 99) WikiText-2 perplexity over 145 chunks lands within
0.006 of the same model's CPU baseline.
Faster fp32-class replacement for the tile_f32 flash-attention inner path on P100
(sm_60). Bit-identical fp32 arithmetic to the un-retiled kernel (same-top 99.28% =
the float-reorder floor; QK differs only in accumulation order, ~1 ulp; P.V
bit-identical), restructured via half2 K/V shared-memory staging that both cuts
shared memory and leans the inner loop.
Measured +4-9% vs the un-retiled fp32 tile kernel, back-to-back. The speedup is a
COMPOUND of two effects the staging produces together (shares not isolated):
(1) occupancy: smem 36992 -> 28800 B/block admits 2 blocks/SM where the un-retiled
kernel fits only 1 (2*28800=57600<=65536; 80 regs would allow 3, smem is the
ceiling); a genuine 1->2 gain, corroborated by a cross-family perf panel;
(2) a leaner inner loop: ~2x fewer QK-loop smem loads, the dropped score round-trip,
one fewer barrier.
__launch_bounds__(...,2) only PINS the target the smem reduction already reaches (a
(...,1) bound compiles bit-identically), so the directive is not itself a lever, and a
3rd block does not help (cost-curve + an implemented regs-80->124 attempt confirm smem
caps occupancy at 2).
Validated on P100 as the drop-in for the carve-out path (pr-p100-fp16). Reviewed by a
6-model Claude council + a cross-family cloud perf panel; the panel corrected an earlier
'not occupancy' framing to this compound one.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On P100 (GP100, sm_60) the fp16 vec kernel used for decode (batch<=8)
accumulates the online-softmax denominator and the P*V product in fp16,
flipping ~3-4% of decode top-1 tokens vs an all-fp32 reference
(llama.cpp#25593). Decode is memory-bandwidth-bound on P100, so routing
sm_60 decode to the in-tree vec_f32 kernel is free (tg128 ~identical).
Gated on cc == CC_PASCAL && Q->ne[1] <= 8 (decode only) inside the
!fp16_mma_available block, so the prefill tile_f16 path, the D=256 prefill
vec path, and fast_fp16_available() are untouched, and the
is_pascal_mla_absorbed_decode early-return (MLA) is unaffected.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ggml_graph_node_has_matching_properties exempted every GGML_OP_CPY node from the source-address check, for both operands. The exemption is needed for KV-cache write copies, whose destination advances each step through the indirect-destination path. It also skips a CPY whose read source (src[0]) moves between graph replays while the surrounding subgraph stays shape-stable, so the captured kernel replays against a stale source pointer and reads the previous step's bytes.
Keep the exemption for the destination operand (src[1]) only, and compare a CPY's read source (src[0]) like any other node. Stable-source copies are unaffected and force no additional re-captures; a CPY whose read source genuinely moves now re-captures instead of reading stale memory.
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
* WIP: indexer_topk on CUDA
* Forgot these
* WIP
* WIP
* This seems to work
* Minor
* Fix bug. Fix suggested by @sayap using GLM-5.2
* GLM-DSA: much better PP long context performance (CUDA)
* DSA: Better way to build the attention mask
* WIP: indexer_topk on CUDA
* Forgot these
* WIP
* WIP
* This seems to work
* Minor
* Fix bug. Fix suggested by @sayap using GLM-5.2
* GLM-DSA: much better PP long context performance (CUDA)
* ggml: add fused sinkhorn op (eps + output-layout params); use it for openPangu mHC
* openpangu: call ggml_sinkhorn directly from mhc_post
---------
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
* IQK AVX2: Replace MM256_SET_M128I(x, x) identity broadcasts with _mm256_broadcastsi128_si256
## Summary
Replace all 132 instances of the identity-broadcast pattern
`MM256_SET_M128I(x, x)` with `_mm256_broadcastsi128_si256(x)` across
8 files in ggml/src/iqk/.
## Correctness
The transformation is bit-exact. The `MM256_SET_M128I(a, b)` macro
expands to:
_mm256_insertf128_si256(_mm256_castsi128_si256(b), a, 1)
which places `a` in the high 128-bit lane and `b` in the low lane.
When `a == b == x`, the result is x replicated to both lanes:
[x_lo: x_hi] = {x, x}
`_mm256_broadcastsi128_si256(x)` produces the identical register state:
it copies the 128-bit input to both lanes in a single micro-op.
Perplexity was verified identical on llama-3.2-1b-Q8_0 and
google-gemma-3-4b-Q4_0-IQ4_XS before and after the change.
The transformation is a pure intrinsic substitution with
zero semantic difference.
## Performance
### Micro-architecture analysis
The old pattern compiles to:
vinsertf128 ymm, ymm, xmm, 1 -- 3 uops, port 5, 3-cycle latency
The new pattern compiles to:
vbroadcasti128 ymm, xmm -- 1 uop, port 5, 1-cycle latency
Both instructions execute on port 5 (Intel), but vbroadcasti128:
- Uses 1/3 the dispatch slots (fewer pipeline stalls)
- Has 3x better latency (1 vs 3 cycles)
- Is not lane-crossing (no bypass delay between 128-bit halves)
### Measured results
#### Test 1: llama-3.2-1b-Q8_0, 2048 ctx, -b 128 -ub 128
Compiler | Metric | Before | After | Change
--------------|--------|---------|---------|-------
MSVC 19.44 | PP t/s | 596.89 | 604.57 | +1.29% (noise imo)
MSVC 19.44 | TG t/s | 55.13 | 55.72 | +1.07% (noise)
Clang 19.1.5 | PP t/s | 645.72 | 700.93 | +8.55% (systematic gain)
Clang 19.1.5 | TG t/s | 54.26 | 54.06 | -0.37% (noise)
#### Test 2: google-gemma-3-4b-Q4_0-IQ4_XS, 4096 ctx, -b 512
Compiler | Metric | Before | After | Change
--------------|----------|----------|----------|-------
Clang 19.1.5 | PP t/s | 262.40 | 314.79 | +19.96% (systematic gain)
Clang 19.1.5 | TG t/s | 30.78 | 32.32 | +5.00% (noise, TG oscilates between 31.5 and 33 t/s before / after)
Clang 19.1.5 | Total ms | 48881 | 44696 | -8.56%
Both tests ran on Intel Core Ultra 265K, 18 threads, flash_attn=1
The IQ4_XS result shows a dramatic PP improvement (+20%) because this
quantization format uses significantly more identity broadcasts in its
dequantization path (lookup-table expansion, scale duplication). The
compressed 4-bit representation requires more setup per block, making
the broadcast-to-256 step a measurable bottleneck that the 1-uop
vbroadcasti128 eliminates.
### Why these 132 instances matter
Every dequantization path (IQ2_XXS through IQ6_K, Q4_0 through Q8_1,
MXFP4) starts by broadcasting a 128-bit lookup table or scale vector
to 256 bits. These are in the inner loop of every quantization format's
dot-product kernel. Reducing each broadcast from 3 uops to 1 uop
cumulatively reduces port-5 pressure across the entire dequant
pipeline.
## Scope
This change touches only the identity-broadcast case (both MM256_SET_M128I
arguments are identical).
* IQK AVX2: Introduce MM256_SET1_M128I(x) wrapper for identity-broadcast pattern
Per Ikawrakow's review: replace direct _mm256_broadcastsi128_si256(x) with
a new macro MM256_SET1_M128I(x) wrapping the intrinsic, so that if the
broadcast turns out harmful on some CPU, only the macro definition needs
changing, not 132 call sites.
#define MM256_SET1_M128I(x) _mm256_broadcastsi128_si256(x)
The 132 identity-broadcast MM256_SET_M128I(x, x) call sites across 8 files
now use MM256_SET1_M128I(x) instead of the raw intrinsic.
* Add --prefetch-experts to stream mmap'd MoE experts into page cache
* Drop fds, fault experts in with MADV_POPULATE_READ instead of pread
* Remove stale note about pread workers
* Move MoE prefetch behind ggml_backend_prefetch_* wrappers
* Cleanup stale comments
* Add --prefetch-experts-threads, drop GGML_MOE_PREFETCH_THREADS env var
When building with clang-cl (MSVC + Clang), the CMake MSVC branch defined
__AVXVNNI__ as a preprocessor macro alongside /arch:AVX2, but clang-cl
requires the actual -mavxvnni target feature flag to enable AVX-VNNI
codegen. Without it, clang-cl refused to inline _mm256_dpbusd_avx_epi32
and _mm256_dpwssd_avx_epi32 into functions compiled under /arch:AVX2,
causing 'requires target feature avxvnni' errors in:
- ggml-quants.c (mul_sum_us8_pairs_float)
- iqk_gemm_iquants.cpp (mul_mat_iq3_xxs_r4_q8_k)
- iqk_gemm_kquants.cpp (mul_mat_q3_k_r4_q8_k)
- iqk_gemm_legacy_quants.cpp (dot, accum_q4_0_quants, operator())
Fix: Detect clang-cl via CMAKE_CXX_COMPILER_ID STREQUAL 'Clang' and
append -mavxvnni to ARCH_FLAGS instead of manual __AVXVNNI__ define.
Also add missing GGML_AVXVNNI handling for the non-MSVC (GCC/Clang on
Linux) branch, passing -mavxvnni as expected.
* CUDA: use the mask tensor stride (nb31) in the tile FA kernels, not ne11
The tile flash-attention kernels (no-tensor-core GPUs, e.g. Pascal/sm_60) indexed
the KQ mask using ne11 (= K->ne[1]) as the row stride. The SWA windowing in
ggml_cuda_flash_attn_ext (the n_swa branch) re-points K/V/mask to the last nton
tokens, setting ne11 = nton while the mask keeps its original row stride nb31.
Indexing by ne11 then reads across mask rows and yields NaN once the window slice
engages (context past nton). Use the mask's own stride nb31/sizeof(half); in the
non-sliced case this equals ne11, so it is a no-op there. Matches how the vec
kernel already computes its mask offset, and how mainline llama.cpp fixed the same
latent bug in its unified tile kernel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* CUDA: apply attention sinks in the tile FA kernels (gpt-oss)
The tile flash-attention kernels took a sinks argument but never applied it, so
gpt-oss (which has a per-head sink logit) got a wrong softmax denominator with
-fa 1 while the -fa 0 path (ggml_soft_max_add_sinks) matched CPU. Apply the sink
after the KV loop, mirroring the vec kernel: the sink joins the running max and
adds exp(sink - max) to the denominator once, rescaling kqsum and VKQ. Only ip==0
adds it so it is counted once across the parallel_blocks KV split (the epilogue
writes the sink-inclusive max/denominator into dst_meta before the combine). The
per-head index is blockIdx.y (the same index the slope uses). Guarded by non-null
sinks, so non-sink models are unaffected. This is the tile-kernel equivalent of
mainline llama.cpp PR #15178; the wmma kernel is left untouched (no Turing/Volta
hardware to validate here, but the same defect likely applies).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On GPUs without FP16 tensor cores (Pascal / sm_60, e.g. Tesla P100) MLA
flash-attention decode falls back to the CPU. The !fp16_mma_available path
routes decode to the f16 vector kernel, whose is_supported check requires
K == V head sizes; MLA's absorbed head sizes are 576/512 (asymmetric), so it is
rejected and attention runs on the CPU. With --cpu-moe that recomputes the full
MLA attention on the CPU every decoded token, which dominates decode at long
context.
Route Pascal MLA decode (Q->ne[1] <= 8 && K == 576 && V == 512) to the f32
vector kernel and enable that kernel for the 576/512 case, including Q8_0 KV.
Scope: decode only (batch <= 8). Prefill (batch > 8) and -fa 0 are untouched;
tensor-core GPUs never reach this branch. Aligned head sizes are byte-identical
(the asymmetric/Q8_0 work folds to a no-op at compile time), so no other model
or configuration is affected.
- fattn.cu: route 576/512 decode to vec_f32 in the !fp16_mma dispatch and its
is_supported mirror.
- fattn-vec-f32.cu/.cuh: accept + instantiate 576/512 (F16 and Q8_0); fix latent
issues exposed by the first asymmetric/large-head use (KQ-row granularity uses
FATTN_KQ_STRIDE not Dv; guard the dst store to tid < Dv; guard the softmax exp
on the KV tail; size Q_i32 by ceil; only convert K/V to F16 when the type is
F16). All are no-ops for the previously-exercised symmetric cases.
- fattn-vec-f32.cuh / fattn-vec-common.cuh: guard the Q8_0 ragged tail (Dk=576 is
144 int32 lanes = 4.5 warps) with the ragged-dim idiom; compile-time-constant
for aligned head dims, so it folds away.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Kawrakow
Joel Farthing
Petr Vilím
Nexesenex
hchengit
mb8565
cora4
Jun Yamog
dmaivel
Lorenzo Vasile