* WIP: indexer_topk on CUDA
* Forgot these
* WIP
* WIP
* This seems to work
* Minor
* Fix bug. Fix suggested by @sayap using GLM-5.2
* GLM-DSA: much better PP long context performance (CUDA)
* DSA: Better way to build the attention mask
* openpangu: Stage-1 converter probe for openPangu-2.0-Flash
Add OpenPanguV2ForCausalLM conversion support (converter-only; runtime graph
is Stage-2). Registers a new LLM_ARCH_OPENPANGU on the Python/gguf-py side:
- gguf-py/constants.py: MODEL_ARCH.OPENPANGU + name, indexer KV keys, 22 new
tensor enums (DSA indexer x4, MoME convs x3, param-sink x2, mHC/Hyper-
Connections x12, block-post-norm), and the full MODEL_TENSORS list reusing
the deepseek MLA + MoE + NextN bricks.
- tensor_mapping.py: arch-specific block mappings that disambiguate the
sandwich norms (post_attention/pre_mlp/post_mlp) and pin every Pangu-only
tensor; non-block global mHC merge module.
- convert_hf_to_gguf.py: OpenPanguV2Model (subclasses DeepseekV2Model) with
set_gguf_parameters (MLA/MoE/indexer/mHC/param-sink/DSA+SWA metadata),
modify_tensors (expert merge, kv_b split, no MTP skip), and the
OpenPanguV2Tokenizer pre-tokenizer hash.
Validated offline against the real 50-shard safetensors index: all 37,587
tensors map to a GGUF target (0 unmapped), and set_gguf_parameters reads only
hparams present in config.json. No weights downloaded; no GPU. Pinned on the
ik/dsa_loop_hadamard_blend DSA substrate.
* openpangu: Stage-2 arch scaffold (LLM_ARCH_OPENPANGU) — loadable, compiles
New arch on main (DSA-decoupled). Declares openPangu-2.0-Flash to the runtime so
the model loads into memory; the compute graph is the next step.
- llama-arch.{h,cpp}: LLM_ARCH_OPENPANGU + name; 3 KV keys (mhc_num_stream,
mhc_recur_norm, param_sink_number); 18 tensor enums (mHC x12, MoME conv x3,
param-sink x2, block-post-norm).
- llama-model.cpp: OPENPANGU tensor-name block, strings matched to the converter.
- llama-model.h: layer + model struct fields (mHC / conv / sink / block-post / merge).
- llama-hparams.{h,cpp}: reader (MLA + MoE + sigmoid gate + indexer + mHC +
param-sink + NextN); n_layer_kv_from_start = n_layer - nextn (MTP skipped).
- llama-load-tensors.cpp: create_openpangu_tensors (GLM-DSA MLA/MoE base + Pangu
tensors; indexer loaded-but-unused for dense fallback); dispatch + is_mla_attn.
Builds clean (CPU-only libllama). Dense-fallback design: no DSA indexer / SWA
windowing / MTP for first generation (exact <=512 tokens). Graph is Stage-2b.
* openpangu: fix compresskv_conv dim (kv_lora_rank, not +rope); pin attention order in spec
* openpangu: end-to-end runtime — build_openpangu graph runs, generates (garbled)
First full forward pass of openPangu-2.0-Flash on ik_llama. Pipeline works end to
end: new LLM_ARCH_OPENPANGU loads the Q4 GGUF, the graph executes, and llama-cli
generates 40 tokens (EXIT=0). Output is currently garbled (tensor-layout bug to
debug), but the structure is proven.
graphs/build_openpangu.cpp: dense decompressed-MHA attention + 4-stream mHC
(Hyper-Connections) with 20-iter Sinkhorn + MoE(sigmoid+shared) + sandwich norms
+ entry stream-repeat/tail-merge + inp_out_ids selection.
Bring-up fixes to load+run:
- llama-vocab.cpp: register 'openpangu' pre-tokenizer (QWEN2 family)
- llama.cpp: OPENPANGU -> LLAMA_ROPE_TYPE_NORM (was defaulting to NONE=-1)
- llama-load-tensors.cpp: full wkv_b load; k_b/v_b as flattened 2D; block_post_norm
dim = S*H (10240); conv weights 2D {3,C}; mHC alpha/beta/gamma + param_sink +
merge params use bare (no-.weight) tensor names
- llama-model.cpp: OPENPANGU is NOT is_mla_attn (decompressed MHA, standard KV cache)
- graph loop bounded to base layers (skip NextN/MTP)
v0 deferrals (need conv-state cache / manual attention path, all documented):
MoME convs (passthrough), o_conv, param_sink. Next: fix the layout bug to coherence.
* openpangu: COHERENT generation — NEOX rope, Sinkhorn orientation, MoME convs, param_sink
Four correctness fixes on top of the end-to-end scaffold, verified checkpoint-by-
checkpoint against a Python golden reference running on the GGUF's own dequantized
weights (block-0 activations now match to rounding at full fidelity):
- rope: NORM -> NEOX. Pangu config rope_interleave=false; the Infer source maps it
as is_neox_style = not rope_interleave (rotary_mode='half').
- mHC Sinkhorn: the flat h_res block is torch-[r,c] row-major, so a bare ggml
reshape lands column-fastest; the doubly-stochastic iteration ran transposed
(Sinkhorn is not transpose-symmetric). One transpose at input fixes the whole
chain including the mhc_post application.
- MoME convs (qa/compresskv/o): were passthrough stubs. Implemented as
out = x + causal_conv1d(x) (every Infer call site uses residual_connection=1;
tap stats confirm the perturbation form). Taps cast f16->f32 for ggml_mul.
Batch-local v0: exact for fresh-sequence prefill; decode steps miss the
t-1/t-2 taps until a conv-state cache exists.
- param_sink: 128 learned latent-KV entries prepended per layer via a manual
attention path (kv_store + explicit soft_max over [sinks ++ cache]); huge
effect at short context. o_conv now applied pre-o_proj on the same path.
flash_attn forced off for OPENPANGU (FA kernel cannot see the sinks).
- converter: add_bos_token=true (HF prepends <|pangu_text_start|> via the
post-processor; the key was absent so ik dropped BOS).
Greedy Q4_K_M smoke, chat template + <think>: coherent CoT reasoning and a
correct answer. Layer-0 instrumentation (opg0_* names) kept for now.
* openpangu: MoME conv-state cache — decode steps get real t-1/t-2 taps
Allocate a per-layer cache_s_l tensor for OPENPANGU base layers holding the last
two pre-conv latents of the three MoME sites, packed
[qa 2*1024 | compresskv 2*512 | o 2*6144] f32 (~60KB/layer). The conv helper
reads the [C,2] history window (zeros at sequence start, kv_head==0), builds
xx = [hist ++ x], and writes the last two columns back each ubatch — the concat
naturally handles both prefill chaining and the T==1 shift. Read precedes write
in graph order; the fixed-offset copy is graph-reuse safe.
Verified: prefill anchors unchanged (bit-path identical, zero-history branch);
-ub 1 token-by-token run matches the golden reference at t4 (qlora_conv 0.084,
R_block 0.008 rel; attn_out 0.15 on one channel = f16 KV-cache rounding, washes
out by post-norm); final logits differ from full-batch only by a common-mode
shift that softmax cancels. Chat-template greedy smoke: think-block repetition
is gone — clean structured CoT and correct answer.
v0 limits documented in the helper: one state slot (single sequence); cache
rewinds leave the state stale.
* openpangu: NextN/MTP speculative decoding — 1.7-1.8x TG on CPU
Wire the three NextN layers (46-48) into ik's MTP speculative framework
(--spec-type mtp). v0 drafts with head 1 (layer 46), self-chained by the
framework.
- llama.cpp: add OPENPANGU to the cparams.mtp arch allowlist (it was silently
zeroed, which left the target context without a logits buffer once the server
enabled embeddings -> GGML_ASSERT(lctx.logits) in speculative_is_compat).
- load-tensors: MTP layers carry no mHC tensors (tail_use_mhc=false in the
reference) — create them only for base layers. nextn.* tensors were already
wired by the Stage-1 probe.
- build_openpangu: extract the attention sublayer into
build_openpangu_attention (shared base/MTP); add build_openpangu_mtp:
eh_proj(cat(enorm(embed), hnorm(prev_hidden))) -> one plain-residual Pangu
block (sandwich norms, convs+param_sink, MoE+shexp, no mHC/block_post_norm)
-> shared_head norm+head. MTP branch returns the draft graph when
mtp_op_type != NONE; main graph keeps all-token outputs under cparams.mtp.
MTP convs run batch-local (no conv-state slot) — affects acceptance only.
A/B (Q4_K_M, CPU, greedy, 192-token chat CoT completion, warm back-to-back,
medians of 3, bracketed B/A/B):
no-spec: 2.44 t/s (2.34-2.86)
--spec-type mtp:n_max=3: 4.23 / 4.49 t/s (brackets) => ~1.7-1.8x
Draft acceptance 34% on CoT prose (46% on repetitive text); spec and no-spec
greedy outputs are byte-identical. Headroom: conv-state for MTP drafts, n_max
tuning, true 3-head chaining (spec_step_idx).
* server: include draft_n/draft_n_accepted in /completion timings
get_formated_timings() (the /completion path) omitted the speculative
counters that get_timings() (the OAI path) already reports; add them,
guarded by n_draft_total > 0 like the OAI path.
* openpangu: position-indexed MoME conv-state ring — rollback-safe spec decoding + MTP draft chaining
The v0 single-slot conv state held the last-2 pre-conv latents of the most
recent batch, so any speculative draft rejection left latents of REJECTED
positions in the state and every later decode ran with wrong t-1/t-2 taps
(3 conv sites x 46 layers). At 192-token greedy runs every spec config
diverged from no-spec, each differently (rejection-pattern dependent).
Replace it with a per-layer ring cache_s_l [n_lora_q+n_lora_kv+n_head*v_dim, 16]:
column pos%16 holds position pos's pre-conv latents ([qa|ckv|o] packed).
Invariant: reads target only positions before the first batch token, which
are committed, and committed latents depend only on the committed prefix -
rollback-safe by construction, no checkpointing. Writes cover the last
min(T,16) batch positions in <=2 contiguous cpy segments; the copy sources
are views of the [hist ++ x] concat so the history read is an ancestor of
every write (read-before-write by graph dependency).
The ring is also allocated for the NextN/MTP layers, so the draft head
chains real conv taps across WARMUP -> sequential DRAFT_GEN steps (was
batch-local zero-history per draft token).
graph_reuse is forced off for the arch: ring view offsets are position-
baked and the reuse patcher only updates the standard K/V-store copies.
Measured cost on the CPU server path: none visible. ggml_set_rows driven
by an input index tensor is the future reuse-safe shape.
Verified (Q4_K_M, CPU, greedy 192-tok chat-CoT, warm single process):
- no-spec output byte-identical to pre-ring build
- spec output byte-identical to no-spec below the n_predict cap, for all
of n_max in {1,2,3,4,6} x p_min in {0,0.3,0.6} (old build: all diverged)
- acceptance n3-p0: 33.9% -> 60.9%; n3-p0.3: 58.1% -> 68.9%
- TG medians: no-spec 3.19-3.32 t/s; mtp:n_max=3,p_min=0.3 6.97 t/s (~2.1x)
* openpangu: DSA lightning indexer + SWA schedule — long-context correctness past the dense fallback
The dense fallback was exact only <=512 tokens (SWA window). This wires the real
DSA/SWA hybrid schedule, self-contained from GGUF keys the converter already
writes (openpangu.swa_layers + sliding_window_list; absent keys keep the old
dense fallback):
- SWA layers (30 base @512): the generic inp_KQ_mask_swa path, per-layer mask
choice in the builder. The NextN/MTP layers are SWA @2048 in the checkpoint
schedule; MTP graphs run in their own context, so the mask fill picks
hparams.n_swa_mtp when built with an MTP op type.
- DSA layers (16, every 3rd): lightning indexer implemented in-graph from the
Infer reference semantics (jointfix _pangu_torch_calib): q_idx = wq_b on the
post-conv post-norm q-lora latent (24x128), k_idx = rms-normed wk(x) shared
across heads, both NEOX-roped on the FIRST n_rot channels; score =
sum_g w_g * relu(q_g . k) in f32, causal-masked, exact top-k via
argsort + ggml_set_rows scatter into a -1e30 base -> additive selection mask
on the existing manual soft_max seam. Selection engages only when the causal
window exceeds index_top_k (2048); below that the layer is exactly dense.
- Indexer keys cached per position (cache_idx_l, f32 [128, kv_size], DSA layers
only) with the same committed-position invariant as the conv-state ring, so
speculative rollbacks stay safe.
- param sinks remain outside both the window and the selection budget, matching
the reference.
Verified (Q4_K_M, CPU):
- <=512 tokens: byte-identical to the dense build (96/160-token greedy)
- indexer scores vs a GGUF-dequant golden reference at 2101 tokens: 1e-3 rel
(f16 weight rounding); top-3 selection indices exact on all compared queries
- >512 coherence clean; 3.4K-token needle retrieval through active selection
(needle outside every SWA window, ~1300 positions pruned) answers exactly
* openpangu: MLA-latent KV cache — attention absorbed into the 512-latent, 14x smaller cache, ~2.2x TG
Store per position only [ckv_norm 512 | roped k_pe 64] (f32, k_l) plus the
transposed 512-latent (f32, v_l, v_trans layout); per-head K/V are never
materialized. q_nope is absorbed through attn_k_b (loaded 2D from the
converter split for base layers; derived at load via llm_prepare_mla for the
NextN layers - now guarded for layers without attention weights, e.g. the
idle NextN heads 2/3). The value side is the latent itself, up-projected
through attn_v_b after the weighted sum, matching the Infer _forward_dsa
reference. param sinks are native latent-space entries, which removes the
per-step full-cache concat+cast that dominated long-context decode.
llama_state row sizes now come from llama_kv_k_row_embd/llama_kv_v_row_embd
(arch-aware), fixing an out-of-bounds crash in the server prompt-cache save
path (hparams-derived 9216-wide rows vs actual 576-wide latent rows).
Verified (Q4_K_M, CPU): layer-0 attention output matches an f32 golden
reference computed from the same GGUF weight encodings (~1e-2 on O(1)
values); MTP spec output byte-identical to no-spec; 3.4K needle retrieval
through active DSA selection exact under greedy. Output differs from the
materialized build at the token level because attn_k_b/attn_v_b are
independently quantized tensors - both are legitimate Q4-fidelity encodings.
Perf (CPU, warm): no-spec TG 3.2-3.3 -> 6.9-7.1 t/s; mtp:n_max=3,p_min=0.3
-> 11.1 t/s (byte-exact, 67% acceptance); prefill 30.5 t/s at 3.4K; KV self
size at 4K ctx: 5.5 GiB -> 391 MiB. Not yet supported on the latent cache:
K-shift/defrag (context shifting) - unreached in current usage.
* openpangu: fence unsupported serving modes, truth-pass comments, drop dead weight/keys
Post-audit hardening. The cache's position-indexed side state (MoME conv ring,
DSA indexer keys) made several generic serving paths silently unsound; they are
now fenced loudly instead of documented as unsupported:
- s_l_position_ring flag on llama_kv_cache: the qnext-state predicate no longer
claims the conv ring, so per-seq state save, seq_cp and the s_copy graph skip it
- state save/restore refused for the arch at every llama_state_* entry (the ring
and idx_l are not in the state format; restoring without them diverges silently)
- K-shift/self-extend assert, defrag skips with a warning, server ctx_shift off
via new llama_model_supports_ctx_shift()
- single sequence enforced at context creation (n_seq_max > 1 refused)
- server prompt-cache reuse limited to pure extension via new
llama_model_supports_partial_kv_reuse(): mid-cache divergence reprocesses from
scratch (the 16-column ring cannot rewind); multi-turn continuation stays fast
- MTP draft length clamped to 13 via new llama_model_max_draft_tokens() so a
rejected draft can never overwrite the ring columns the next decode reads
- K/V cache types forced to f32 for the arch so the KV size log reports the truth
- cache_size(): real latent-cache branch (was falling through to the ~14x larger
materialized estimate used for offload planning)
- unused fused wkv_b no longer loaded (TENSOR_SKIP; the graph runs entirely on the
pre-split k_b/v_b), llm_prepare_mla openPangu special-case removed (it was a no-op)
- stale v0 comments rewritten to describe the shipped graph; converter stops
writing dead keys (dsa_layers, block_post_layernorm_idx) and the tokenizer
pre-hash is registered in convert_hf_to_gguf_update.py
Gates on this build: greedy spec output byte-identical to no-spec (EOS-terminated,
sha-equal); 3.4K needle retrieved exactly; -np 2 / state save / n_max=20 / stale
prefix reuse all refused or clamped with clear messages.
* openpangu: assert kv_head == first batch position at graph build
The ring, indexer and latent stores are addressed by absolute position through
kv_head; the fences make append-only decode the only reachable mode, but the
invariant was unchecked. Assert it at both graph entries (base and MTP) so any
future cache plumbing that breaks it fails at build instead of corrupting
output. Worst-case measurement builds pass pos = null and are exempt.
* openpangu: cont h_pre before the mHC broadcast mul (CUDA binbcast misreads strided views)
h_pre is a row-slice view of the fused mixes tensor. The CPU mul handles the
strides; the CUDA broadcast path reads the view as if contiguous, so token 0
mixes correctly and every later token gets h_post/h_res rows instead. First
divergent node in the whole graph (oracle rel 0.36 at opg0_attn_mhcpre_x,
fixed to 7.5e-5). Sibling views h_post/h_res were already cont-wrapped, which
is why only h_pre was exposed.
* openpangu: keep DSA zero-trick sources finite (CUDA clamp propagates the 0*(-inf) NaN)
The selection-mask base and zeros were built by scaling the MASKED scores by
zero, but post-mask sc contains -inf and 0 * -inf = NaN. The CPU clamp launders
NaN back to -1e30 (fminf/fmaxf ignore NaN); the CUDA clamp propagates it, so
every DSA layer emitted NaN masks at n_kv > top_k and logits collapsed
(observed: eval-callback CLAMP sum -1.3e36 on CPU vs nan on CUDA, 11748 NaNs
downstream). Scale the pre-mask finite scores instead, which is correct on any
backend regardless of clamp NaN semantics. Also defensively cont the strided
KQ_mask slice feeding the score add (same strided-view kernel class as the mHC
h_pre fix; unproven here but cheap). Gates after fix: 2600-token probe coherent,
3.4K needle exact ('7391') with and without MTP speculation, PP ~120 t/s.
* openpangu: f16 latent KV cache option (explicit -ctk/-ctv f16 halves cache memory, f32 stays default)
Track explicit cache-type requests through CLI/env; openPangu resolves no-request
to f32 (unchanged), accepts explicit f32/f16, warns and falls back to f32 for
BF16/quantized. Sink and cached-token KQ paths stay separate until after KQ so
the latent cache is read directly without the f32-only concat; value is the sum
of the sink and cache matmuls. Ring and DSA indexer caches stay f32; cache_size()
follows the resolved types.
* openpangu: enable graph reuse
* openpangu: wire multi-head MTP drafting
* openpangu: add MTP heads override
* openpangu: keep MTP update logits last
* openpangu: scope MTP warmup heads
* speculative: apply per-request MTP heads before warmup
* openpangu: fix multi-head MTP warmup computing on unwritten inputs
Each chained head called the build_inp_* helpers itself, so the warmup and
update graphs held one inp_tokens/inp_pos/inp_out_ids/KQ_mask tensor per
head while llama_set_inputs only fills the tensors the lctx pointers
reference, i.e. the last head's copies. Every head but the last read
unwritten compute-buffer memory: with heads=3 active even head 1's ring,
latent cache, and cached one-token draft were computed from garbage, which
is why depth-1 acceptance measured 4% against 98% for the heads=1 control.
Create the batch inputs once in build_openpangu and pass them to every
build_openpangu_mtp call, and fix the two chaining errors that were hiding
behind the garbage inputs:
- Shift the chained hidden: head k+1's row at position p consumes head k's
output row at p-1, the same convention head 1 uses for the target's
conditioned hidden rows. The predecessor of a batch's first row lives in
the previous warmup/update, carried across decodes through a new
inp_mtp_carry input backed by lctx.mtp_carry (written back per ubatch,
zeroed when a prompt warmup restarts from position 0).
- Fill head 3's cache row at draft step 2: each draft step runs one head,
so head 3's own decode at step 3 attended over a never-written row at
the step-2 position. Pre-write it from the committed carry.
Also include the active head count in the graph-reuse key next to the
existing step index (reuse stays forced off for this arch).
* speculative: default MTP drafting to a single head
A stage without an explicit heads= override previously resolved to 0,
meaning all model heads, so multi-head drafting was silently on by
default for models that carry more than one NextN layer. Keep it opt-in
(heads=N or heads=0 for all) until multi-head measures a win over the
single-head config; single-head models are unaffected either way.
* speculative: fence MTP head upshift over a warmed prefix
Deeper NextN heads only hold valid cache rows for spans that were warmed
with them. A request drafting with more MTP heads than the cached prefix
was warmed with (e.g. a heads=1 conversation continued with heads=3, a
pure extension the divergence fence deliberately allows) would read
never-written deeper-head rows: verification keeps the output correct,
but acceptance quietly collapses and any measurement taken there is
misleading.
Track the minimum head count the committed context has been warmed with
since position 0 and have the server reprocess from scratch when a
request asks for more. Also announce the model's NextN head count and
the single-head default once at MTP context setup.
* openpangu: skip dead MTP chain compute and stall-free carry readback
The update chain's last head and the draft-time row fill only matter for
their latent-cache and conv-ring writes; their FFN, norms, and shared
head fed nothing. Add a cache-writes-only mode to the MTP block builder
that returns after the attention block, and use it at both sites.
The multi-head carry readback previously synchronized the scheduler
after every warmup/update decode, a hard stall on CUDA. Issue the
device-to-host copy async on the backend stream instead (stream order
protects the source buffer from later graphs) and synchronize lazily
when the host buffer is next consumed or resized.
* openpangu: stop emitting fused kv_b tensor
* openpangu: default latent cache to f16
* openpangu: refuse unsupported latent cache types
* Window OpenPangu SWA cache reads
* Gather OpenPangu DSA decode reads
Gather DSA decode attention over the selected latent rows for OpenPangu base-model decode and verify graphs. The gathered branch now uses ggml_top_k order directly, runs maskless softmax over sinks plus selected rows for T <= 14, and derives values from the gathered k_l rows instead of the transposed latent cache.
* Chunk OpenPangu indexer prefill scoring
* Chunk OpenPangu prefill attention
* Gather OpenPangu sparse prefill attention
* Drop OpenPangu value cache
* Add OpenPangu indexer cache type flag
* Add OpenPangu q8_0 latent cache type
Store the OpenPangu MLA latent K cache as q8_0 via -ctk q8_0 (about 0.53x of
f16); the default stays f16 so behavior is unchanged without the flag. Latent V
stays f16/f32.
The q8 latent cache is a storage format only: it is dequanted to F32 before all
compute. K reads go through openpangu_build_k_latent_for_read, V derivation
through openpangu_build_v_latent_from_k (full 576-wide row to F32, then slice),
and the DSA gather paths already dequant via get_rows. Feeding a q8 latent view
directly into the KQ mul_mat corrupts large-context prefill, so that path is
removed for quantized caches. The cache write stages ckv and kpe through F32 and
writes one full 576-wide q8 row per token.
Verified on a small discriminator model: the default f16 path is byte-identical
to the prior code; the first-DSA-layer attention envelope is within 0.6% of the
f16 cache (linf_rel 0.0057); top-k selection is bit-identical between cache
types; the q8 latent cache is 0.531x the f16 size at 8K and 32K context; and
generation stays coherent on both the dense and DSA-gather paths at all tested
context lengths.
* Remove OpenPangu debug trace env knobs and redundant DSA_TOPK override
Drop the five LLAMA_OPENPANGU_*_TRACE debug-logging knobs (DSA_GATHER_TRACE,
IDX_CHUNK_TRACE, ATT_CHUNK_TRACE, PREFILL_GATHER_TRACE, SWA_WINDOW_TRACE) and the
LLAMA_OPENPANGU_DSA_TOPK override, which duplicated the -dsatk / --dsa-top-k CLI
flag; top-k now comes solely from cparams.dsa_top_k. The five perf-tuning knobs
(DSA_GATHER, IDX_CHUNK, ATT_CHUNK, ATT_KQ_MAX_MIB, PREFILL_GATHER) are retained
pending the perf battery. No change to default behavior.
* Subchunk OpenPangu DSA prefill gather to fit CUDA grid limit
The prefill gathered-attention ggml_get_rows produced dst rows = topk *
token_chunk (2048 * 256 = 524288) mapped to the CUDA grid.y dimension, which
caps at 65535, crashing with GET_ROWS invalid argument at long context (N_KV
around 10.5K with the natural topk of 2048). Split the prefill gather into token
subchunks so topk * subchunk_tokens stays within the grid limit, and guard the
decode gather with the same fit check (falling back to the dense masked path if
a pathological topk would not fit). The subchunking is over the token dimension
only, so per-token attention is unchanged and the result is numerically
identical. Verified: the GPU sweep runs past the old crash boundary to 22K+ with
zero CUDA errors; CPU and -ctk q8_0 paths unaffected.
* openpangu: fix scheduler node budget for chunked DSA prefill; drop unused attn_kv_b; remove env tunables
- Size the scheduler graph node budget for the chunked DSA prefill so 32K/ub2048 no
longer trips the hash-set reservation assert; derive the extra budget from the
builder's chunk/top-k/window structure with a fixed safety margin.
- Remove LLAMA_OPENPANGU_* environment tunables from both the node-budget estimator
and build_openpangu.cpp; use fixed constants in both so they stay in sync.
- Converter: emit only the split attn_k_b/attn_v_b projections and drop the unused
fused attn_kv_b tensor.
* openpangu: restore DeepSeek converter kv_b; drop trace env + dead code; fix dense-fallback node budget
- convert_hf_to_gguf.py: restore fused attn_kv_b in DeepseekV2Model (shared
parent); openPangu subclass keeps split-only k_b/v_b. Stops newly-converted
DeepSeek GGUFs from failing to load.
- src/llama.cpp: remove LLAMA_GRAPH_REUSE_TRACE getenv, hit/miss counters, and
the unconditional destructor log (no getenv or behavior change for any arch);
node-budget estimator now covers the dense-fallback (n_swa==0) attention-chunk
loop while skipping absent idx/top-k terms, preserving a strict overcount;
remove unreachable openPangu split-cache block.
- src/llama-context.h: drop now-dead graph_reuse_hits/misses members.
- include/llama.h: move type_k/type_v/idx_type_k *_explicit bools to struct end
to avoid a mid-struct ABI shift for out-of-tree consumers.
- src/graphs/build_openpangu.cpp: replace vestigial env-struct singletons with
the OPENPANGU_* constants; drop a redundant Sinkhorn permute round-trip
(one transpose; greedy output verified byte-identical).
Decode output unchanged (byte-identical greedy generation verified); shared-file
changes are openPangu-gated or restore the pre-PR baseline.
* openpangu: chat-parser support (reasoning split + thinking toggle)
Two openPangu-only fixes, both gated on the arch-unique token
<|pangu_text_start|> so no other model's parsing changes.
- chat-diff-analyzer: add a workarounds entry that force-sets TAG_BASED
reasoning with an empty start and a </think> end. openPangu prefills
<think> in the generation prompt, so the output is delimited only by
</think>; the differential detector otherwise learns start="<think>"
from the assistant-history form and fails to split, leaking reasoning
into content. Same shape as the existing Laguna prefill patch.
- chat.cpp: bridge enable_thinking to the template's `thinking` variable.
openPangu's template gates reasoning on `thinking` rather than the
ecosystem-standard `enable_thinking`, so the standard toggle was inert.
An explicit `thinking` chat_template_kwarg still overrides via the
extra_context merge.
Blast radius: test-chat-auto-parser 437/437 unchanged; the sole
test-chat-template diff is a pre-existing GLM trailing-newline.
* openpangu: use ggml_cast for latent dequant reads
Replace ggml_cpy(view, ggml_new_tensor_2d(F32, ...)) with ggml_cast in the MLA
latent V-from-K and K-read helpers. ggml_cast emits the identical GGML_OP_CPY
node into a fresh f32 tensor, so behavior is unchanged; it is the idiomatic
form. Per review.
* openpangu: narrow SWA reuse-key fields to 32-bit
The openpangu_swa_window_view reuse key stored n_kv/n_tokens/window/pad as
int64_t, but these are bounded well under 2^31 (window/pad are uint32_t at
source; n_kv/n_tokens <= context length). Narrow to int32_t/uint32_t and drop
the widening casts. w_view/win_off stay int64_t: they feed ggml view
dims/offsets. Per review.
* openpangu: precompute param_sink derived tensors at load
The per-layer attention-sink block (sink_blk [576,NS]) and its transposed
latent (s_lat_t [NS,512]) are pure functions of the layer weights, yet were
rebuilt every eval across all 49 layers (RMS-norm + cast + concat + transpose).
Compute them once at load, mirroring the wk_b derived-weight precompute, and
read the stored tensors in build_openpangu_attention. Numerically identical;
removes per-token work at decode.
* openpangu: replace conv position-ring with ggml_ssm_conv + spec-rollback checkpoint
Migrate the MoME depthwise causal conv (three sites per attention sublayer:
qa-lora, compressed-kv, attn-out) from the bespoke 16-column position-indexed
ring onto the core ggml_ssm_conv op with a recurrent conv-state slot.
Cache: s_l becomes [2*conv_col_ne, qnext_state_slots], holding the (d_conv-1)=2
history taps per channel for the three sites (float offsets 0 / 2*n_lora_q /
2*(n_lora_q+n_lora_kv)). Drops the conv_hist_idx / conv_write_idx graph inputs
and their fill in llama_set_inputs; adds one single-sequence sq input for
ggml_ssm_conv shared across the three sites and the MTP head.
Speculative rollback: the position ring self-healed rejected draft columns by
absolute position; a recurrent slot does not, since seq_rm is a no-op for
recurrent state. openPangu is admitted at the three spec-checkpoint save/init
gates so the whole-slot shadow checkpoint (gpu-fallback) snapshots the conv
slot before drafting and restores it before the accepted-token replay. The
restore path is already keyed on ckpt.valid, so no gate change is needed there.
Per-step checkpoint mode is declined for openPangu, which has no SSM recurrent
term, so auto mode resolves to the whole-slot shadow.
Gated: non-spec needle unchanged; MTP-spec needle correct with healthy draft
acceptance (rollback verified via the acceptance canary).
* openpangu: single ggml_concat copy for the latent cache store
The non-quantized latent store split the [ckv | roped k_pe] row into two views
and two cache copies, with a base_offset field on the CacheCopy struct to place
the second one. Match the quantized path: concat the two parts and do one copy
into the cache row. This drops the second cache-copy slot (OPENPANGU_COPY_K_KPE)
and removes base_offset from CacheCopy entirely.
Cache contents are unchanged: the concat writes the same [ckv 512 | k_pe 64]
bytes to the same row. Gated on the needle for both the f16 latent path (the one
that changed) and the q8 latent path, plus coherence.
* openpangu: reuse the shared kr_l indexer cache instead of a separate idx_l
The DSA lightning indexer stored its per-position keys in an openPangu-only idx_l
cache, parallel to the kr_l indexer cache GLM-DSA already uses. Both have the same
storage contract: [indexer_head_size, kv_size], idx_type_k dtype, one row per KV
cell, written at kv_head and read [dim, n_kv] from zero. openPangu now allocates
its indexer keys into kr_l and shares the dsa_cache_copies graph-reuse fixup.
The fixup patch is factored into a helper that both the generic path and the
openPangu update_cache_copies branch call, so the openPangu indexer copy is
repointed to the current kv_head on graph reuse like every other cache write.
This drops the idx_l vector, its allocation and memory accounting, and the
openPangu third cache-copy slot (now one latent copy per layer).
Per-arch allocation predicates stay separate (GLM uses indexer_is_full, openPangu
uses the window==0 DSA schedule); only the kr_l storage and the copy fixup are
shared. openPangu keeps its no-shift/no-defrag/no-state-I/O behavior, and the GLM
Hadamard/k-shift logic stays GLM-gated.
Gated: needle correct on f16 and q8 latent caches and under MTP speculation
(acceptance unchanged at 0.67), plus coherence.
* openpangu: discard pos-0 graphs from reuse; retire stale conv-state comments
The ggml_ssm_conv refactor bakes the pos-0 conv-state reset into graph
topology (a scale-by-zero node on the state view). A graph built at pos 0
could be reused at pos > 0 when the batch shape and padded n_kv match (a
1-token prompt followed by TG is the concrete case), zeroing the conv
history on every reused decode. Admit openPangu at the existing
reset_previous gate so pos-0 graphs are discarded from reuse, the same
guard the qnext recurrent state relies on.
Also retire the internal phase-plan comments the conv refactor left
behind: they claimed the spec-checkpoint wiring had not landed in the
commit that landed it, and misdescribed the s_l slot as awaiting rollback
support.
Gated: needle 8457 on f16 and q8 latent, MTP-spec needle (drafts fully
accepted), coherence.
* openpangu: drop the _explicit cache-type plumbing; validate unconditionally
Review follow-up (item 1 of the second review). The explicit/default
distinction carried less than claimed: the latent K/V fallback was f16,
which is already the -ctk/-ctv and API default, so distinguishing unset
from set-to-the-default bought nothing, and the two bools were behaviorally
redundant. The only load-bearing use was the indexer cache, where openPangu
defaulted to f32 while -ictk defaults to f16. Gating the f16 indexer
directly (needle on f16 and q8 latent paths, MTP speculation, coherence)
shows no quality difference, so openPangu now takes the standard f16
indexer default and the f32 special case is gone. Default indexer cache
memory halves (64 -> 32 MiB at c 8192).
Removes type_k_explicit/type_v_explicit/idx_type_k_explicit from llama.h,
the cparams/mparams plumbing, and common; the resolve helpers become plain
unconditional validators, so -ctk q8_0 is honored and an unsupported type
errors out at load instead of silently coercing.
Gated: needle 8457 on the new f16-indexer default, on q8 latent with MTP
speculation, and with -ictk f32 explicitly honored (64 MiB f32 buffer in
the load log); -ctk q4_0 and -ictk q4_1 refused with a clear error.
* openpangu: keep MTP draft decodes position-contiguous under speculation
The MTP framework's one-token draft shortcut caches a prediction one row
past the accepted prefix during the accepted-token update, then skips
re-decoding the last sampled token at the next draft round. A
mask-addressed cache tolerates the resulting position gap; openPangu's
position-addressed append-only cache (cell == position) does not: after a
rollback the next draft decode lands one cell behind its position, and
after a full acceptance the cache head sits one row ahead of the next
draft base, either way tripping the kv_head == pos[0] invariant and
aborting the server. The checkpoint admission in the conv refactor made
this the standard openPangu speculative flow; the needle-first gates
never generated enough draft rounds against a short prompt to reach it.
Decline the shortcut re-seed for openPangu in mtp_accept_batch (restoring
the drafting behavior all measured acceptance numbers were taken on) and
trim rows at or beyond the draft base in mtp_speculative_gen_draft, so
every draft decode stays position-contiguous with the cache head.
Gated: the crashing flow (short prompt, 512-token spec generation, then a
second request) completes with acceptance 0.60 prose / 0.87 code,
matching the pre-checkpoint baseline profile; needle 8457 plus coherence
on f16+spec and q8+spec.
* openpangu: remove stale ring limits and fix MTP graph reuse
* cli: preserve speculative carry on fallback
Decode an already-emitted pending token when a draft cannot be used instead of sampling unchanged logits and duplicating output. Document single-head MTP as the default and multi-head modes as experimental.
---------
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
* Add GLM-5.2/DeepSeek-V3.2 DSA lightning indexer (batch-local, single-seq prefill)
Implements the sparse top-k "lightning indexer" attention for LLM_ARCH_GLM_DSA
in build_deepseek2_layer_attention (ik's deepseek2 graph).
What it does (per layer, gated on model.arch==GLM_DSA && indexer_attn_q_b):
- indexer_q = indexer_attn_q_b(q_lora latent), split rope(64)/nope(64), NEOX-rope
the pe part, concat. indexer_k = indexer_attn_k(attn_norm out), LayerNorm w/ bias,
same rope/concat (single key head, MQA).
- scores = relu(indexer_k . indexer_q), scaled per-head weights (indexer_proj),
summed over heads, + base causal mask, then ggml_top_k(min(top_k, n_tokens)).
- sparse mask: ggml_fill(-inf) -> ggml_set_rows(0) at top_k positions -> + causal,
used in the soft_max_ext attention path (-mla 1 -fa 0) instead of KQ_mask.
Simplifications (intentional, proven sound):
- Batch-local: no indexer KV-cache. Indexer keys are the current batch tokens.
- Walsh-Hadamard transform omitted: orthonormal rotation, (Hq).(Hk)==q.k, no score change.
Validation (GLM-5.2-UD-IQ2_M, 3x P100, -mla 1 -fa 0):
- Compiles clean (CUDA sm_60); loads and runs.
- c512 -b512 (n_seq=1) PPL = 2.7760, byte-identical to dense baseline (indexer
disabled) = 2.7760, all 8 chunks match -> indexer is an exact no-op when
top_k>=n_tokens. Proves correctness-preservation.
- 3105-token prompt completion (top_k=2048 < 3105 -> indexer ACTIVELY masks):
prompt-eval produces coherent, accurate continuation, identical to dense for the
prompt+early-gen tokens. No NaN/crash. Confirms the masking path works in prefill.
Known limitations (documented follow-ups, NOT handled):
- Single-sequence prefill only. Multi-sequence batches (n_seq>1, e.g. perplexity
default n_batch>n_ctx) and kv_head>0 (decode) break the batch-local key->slot
mapping. n_seq>1 -> NaN (use n_batch==n_ctx). Decode (kv_head>0): each generated
token sees only itself as an indexer key, so generation degenerates into repetition
after the prompt (dense A/B stays coherent) -- this is the decode-cache stub, the
documented next step.
- Flash-attn path (-fa 1, F16 mask) still uses dense KQ_mask (soft_max path only).
- Decode indexer KV-cache + Hadamard cached-K storage not implemented.
Runtime gate: DSA_INDEXER_DISABLE=1 falls back to dense attention (for A/B).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: decode-correct via persistent indexer-K cache
Make the lightning-indexer correct for DECODE (not just prefill). Previously the
indexer was batch-local, so a generated token only scored against itself and
generation degenerated. Now the indexer keys are cached across the full context.
Changes
- llama_kv_cache: add per-layer indexer-key cache `kr_l` [indexer_head_size, kv_size]
(F16, MQA single head), allocated alongside the MLA latent cache for GLM_DSA.
- build_deepseek2_dsa_indexer: write the batch's (Hadamard-rotated) indexer keys to
kr_l at kv_head, read back the full [128, n_kv] cached keys, and score the indexer
queries against ALL past keys. Returns the full descending argsort of the scores.
- Walsh-Hadamard rotation of indexer q/k (cparams.dsa_indexer_hadamard, default on;
filled in llama_set_inputs). Score-preserving; improves cached-K F16 precision.
- build_deepseek2_dsa_sparse_mask: rank-based full-coverage scatter (write a 0/-BIG
penalty into EVERY key slot keyed by rank) instead of partial set_rows into a -inf
fill — the CUDA in-place set_rows does not preserve an un-written base, which had
corrupted decode when n_kv > top_k.
- Attention-sink force-inclusion (DSA_SINK, default 1): boost the first key(s) so the
sink always survives top-k. The IQ2_M-quantized indexer under-ranks the sink, and
masking it collapsed decode; with the boost, top_k=2048 over n_kv>2048 stays coherent.
ggml backend fixes (needed by the indexer)
- CUDA argsort: report unsupported when padded ncols > 1024 (one-thread-per-column
bitonic launch limit) so the scheduler falls back to the CPU argsort. Fixes
"invalid configuration argument" for top_k over a large n_kv.
- CUDA cpy/dup: support I32 -> I32 (top_k index copies / cross-backend moves).
Validation (GLM-5.2-UD-IQ2_M, 3xP100 + --cpu-moe, -mla 1 -fa 0)
- c512 PPL = 2.0743, byte-identical to dense (all 8 chunks): no-op path exact.
- Short-context decode (300 tok): coherent, identical to dense.
- Long-context decode (2521-tok prompt, n_kv>top_k, real masking of ~474 keys,
120+ tok generated): coherent with the sink boost; dense A/B also coherent.
Gated behind arch==GLM_DSA + indexer tensors + kr_l cache; DSA_INDEXER_DISABLE=1
forces dense. Remaining: FA path still uses the dense KQ_mask; multi-sequence
(n_seq>1) batches; deepseek32 arch wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: wire sparse mask into the flash-attention path (-fa 1)
The DSA sparse top-k mask is now applied on the -fa 1 path (our serving config),
not just -fa 0 soft_max. c512 PPL on -fa 1 = 2.0743, byte-identical to dense
(no regression, indexer no-op exact at n_kv <= top_k). Gated arch==GLM_DSA with
DSA_INDEXER_DISABLE escape; -fa 0 path unchanged.
Long-context -fa 1 decode coherence (n_kv > top_k, mask actually biting) validation
is still running at commit time; the FA mask reuses the same full-coverage scatter
proven coherent on the -fa 0 decode path, so it should hold, but confirm before
relying on long-context -fa 1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: UPDATE 4 — MLA-FA fix merged, FA path validated, multi-seq characterized
Document the re-validation after cherry-picking the MLA-FA vec-decode fix (5f18dcc0):
- FA path is ALIVE. Long-ctx -fa 1 decode (2521-tok prompt > top_k, mask actively
biting) is now COHERENT at -mla 1 and -mla 3, vs the pre-fix degeneration into
"0.0.0.0..." repetition. Matches dense (DSA_INDEXER_DISABLE) and -fa 0 controls.
- c512 -fa 1 PPL: indexer-ON == dense == 2.0854, byte-identical all 8 chunks (exact
no-op when n_kv <= top_k; no regression). The 2.0743->2.0854 shift is the MLA-FA
fix changing V accumulation, not an indexer artifact (ON==dense proves it).
- Indexer is feature-complete + validated for single-seq prefill+decode on both
-fa 0 and -fa 1, at -mla 1 and -mla 3 (the R740 serving target).
Remaining PR gaps, characterized honestly:
- Multi-seq (n_seq>1) with active mask is BROKEN (n_seq=2 c4096 PPL 62.6 vs dense
multi-seq 2.54 and single-seq indexer 3.05). No NaN/crash anymore. Root cause:
the indexer uses a single scalar kv_head/n_kv for the whole ubatch; multi-seq
needs per-sequence cache writes + per-sequence top-k. Fix deferred (structural).
- deepseek32 arch: N/A in this fork. DSA lives entirely under LLM_ARCH_GLM_DSA;
there is no LLM_ARCH_DEEPSEEK32 enum. Documented the steps to add one if a real
deepseek32 GGUF is ever served.
Also commit DSA_REFERENCE.md (verbatim mainline deepseek32/glm-dsa source, the port
reference), trimmed of a stray agent-handoff footer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: per-sequence attention sink — fix multi-seq (n_seq>1)
UPDATE 5. The DSA lightning indexer was numerically broken for multi-sequence
batches once the top-k mask bites (n_kv > top_k): c4096 n_seq=2 PPL 62.6 vs
dense 2.54, while single-seq was fine. Root cause: the attention-sink
force-include boosted the GLOBAL key range [0, n_sink) by +1e20, which only
protects sequence 0's sink. With several sequences packed contiguously into one
ubatch (seq 0 at cells [0,n0), seq 1 at [n0,n1), ...), every non-first
sequence's sink lives at cell n0.. (not cell 0), got no boost, and was dropped
from top-k once the mask bites — collapsing that sequence (chunk[2]=61.2 while
chunk[1]=2.33).
The cache write and score/argsort were already per-sequence correct: tokens are
placed contiguously like the main K cache, and the base KQ_mask (filled from
kv_self.cells[i].has_seq_id) already drives cross-seq keys to -inf before
argsort. Only the sink was anchored at the wrong (global) cell.
Fix: replace the global arange sink boost with a per-graph input tensor
inp_dsa_sink {n_kv, n_tokens} (F32), filled on the CPU in llama_set_inputs from
kv_self.cells exactly like the KQ_mask:
inp_dsa_sink[j,i] = 1e20 iff cell[i].pos in [0,n_sink) AND
cell[i].has_seq_id(seq_of_query_j), else 0
so each query force-includes only its OWN sequence's sink. For a single
contiguous sequence from pos 0 this is exactly the old "cell index < n_sink"
set with the same magnitude, so n_seq==1 is byte-identical.
Validation (3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1, wikitext-2):
- c4096 n_seq=2 indexer chunk[2]: 61.2 -> 3.07 (== single-seq 3.05).
- c2048 topk=1024 (mask bites): n_seq=4 == n_seq=1 chunk-for-chunk
(2.5005/2.6080/2.7759/3.1137 vs .../3.1138) -> multi-seq is numerically
identical to processing each sequence alone.
- c512 n_seq=1 indexer ON == dense, all 4 chunks byte-identical (no regression).
n_seq=4 at full c4096 (n_kv=16384) OOMs the P100 compute buffer (capacity, not
correctness; n_seq=4 proven correct at c2048/n_kv=8192).
GLM-5.2 DSA indexer is now sequence-correct for n_seq>=1, prefill+decode,
soft_max+FA, -mla 1/-mla 3. Fully general and PR-ready.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: UPDATE 6 — serving-correctness (kr_l maintained across shift/defrag/seq-ops; per-seq sink on first-present pos)
An adversarial review found the indexer was proven on the perplexity path but
not the serving path: the persistent indexer-K cache kr_l was written/read but
never *maintained* by the KV-cache mutators, and the attention sink anchored on
absolute pos<n_sink (wrong after multi-turn seq_rm). This closes those gaps and
pins down what is actually reachable on the MLA model.
kr_l maintenance:
- build_k_shift (llama-build-context.cpp): rotate the indexer keys by the same
per-cell delta as the main K. The cached key is H*concat(RoPE(k_pe,pos),k_nope),
so un-Hadamard (H sym/orthonormal => H*H=I) -> RoPE-delta the pe sub-block ->
re-Hadamard. Exact because GLM-DSA has no rope-scaling metadata (ext_factor=0,
attn_factor=1, freq_scale=1), so NEOX RoPE is pure/composable. Params mirror the
forward indexer RoPE exactly (rope_factors=nullptr); no DEEPSEEK2 yarn-shift leak.
Non-in-place (cont->rope->concat->re-Had->cpy), no aliasing. K-shift Hadamard
input filled in llama_set_k_shift with the identical Sylvester construction.
- build_defrag: kr_l row-move mirrors the k_l move (defrag never changes pos, so
no re-RoPE). max_moves divisor 6->9 *n_layer when the indexer cache is present.
- seq_rm/seq_cp/seq_keep are metadata-only (verified) so kr_l rows stay matched to
cells; seq_add/seq_div set has_shift and route through K-shift. No seq-op change.
Per-seq sink (llama.cpp llama_set_inputs): anchor on each sequence's FIRST PRESENT
pos (min present pos over the scored n_kv span), not absolute pos<n_sink. After
multi-turn seq_rm drops a sequence's early tokens its earliest survivor has
pos>=n_sink; the absolute test would protect nothing. Fresh seq at pos 0 => min=0
=> byte-identical to the old behaviour.
Serving-shift finding (the whole point): a RoPE context-shift on this model is
REFUSED BY THE ENGINE. get_can_shift() returns false for all MLA models
(is_mla_model() includes GLM_DSA); llama_kv_cache_update returns 1 ->
"main : failed to eval". Reproduced AND isolated with a dense control
(DSA_INDEXER_DISABLE=1): dense fails identically at the same token. The failure is
pre-existing MLA engine behaviour, independent of the indexer. On the MLA path the
shift never happens, so the indexer's kr_l can never desync via K-shift; the
build_k_shift kr_l block is correct-and-dormant (documented loudly in code).
Validation (3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1, GGML_CUDA_NO_PINNED=1,
numactl --interleave=all, wikitext-2):
- No regression: c512 n_seq=1 indexer ON == dense == 2.1957 +/- 0.12031,
byte-identical all 4 chunks (2.2770/2.8741/2.3956/2.1957).
- Multi-seq: c4096 n_seq=2 chunk[1]=2.33 chunk[2]=3.07 healthy (== UPDATE 5;
per-seq sink change did not regress).
- Serving shift: engine-refused for MLA, dense control fails identically.
- Independent adversarial review: GO, no correctness defect in the diff.
- Build clean (llama-cli, llama-perplexity, sm_60).
Comments updated (build_deepseek2.cpp): multi-seq+FA no longer limitations; sink
description matches per-seq min-pos anchoring; BIG=1e30 masks on both soft_max and
FA paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-5.2 DSA indexer: UPDATE 7 — FIX latent graph-reuse cache-fixup omission for the kr_l indexer cache
update_cache_copies() re-points the K/V cache writes to the current kv_head whenever a
compute graph is REUSED (can_reuse_graph reuses iff kv_self.n == prev->n_kv). The persistent
indexer-key cache write (kr_l) is a separate ggml_cpy whose destination view bakes kv_head at
graph-build time, and it was NEVER registered for that fixup. Under FA the cache pads to 256,
so consecutive single-token decode ubatches share the same padded n_kv and the graph IS reused;
without the fixup the kr_l write keeps landing in the first ubatch's slot and later ubatches
never populate their own recent index-key cells (those cells stay at the alloc-zeroed 0.0).
Structurally identical to the MiniMax MSA bug (fork commit 133d14c9).
Fix (mirrors the K/V cache_copies fixup, same shape as MSA 133d14c9):
- llama-context.h: new std::vector<CacheCopy> dsa_cache_copies.
- llama.cpp ctor: resize dsa_cache_copies to n_layer (null entries -> no-op when DSA off).
- build_deepseek2.cpp: register the kr_l ggml_cpy as dsa_cache_copies[il] = {kr_cpy, kr->nb[1]}.
- llama.cpp update_cache_copies(): re-point each registered cpy view_offs = kv_head*step and
patch src[1]->data/data, exactly like K/V, with the c.cpy->view_src == kv_self.kr_l[il]
(+ null/op) guard the MSA fix omitted. soft_max / non-DSA paths byte-identical.
Validation (GLM-5.2-UD-IQ2_M, 3x P100 -ngl 99 --cpu-moe -t 32, NO_PINNED, P2P-disable patch
re-applied to get a working multi-GPU baseline — see UPDATE 7.3; that patch was lost in the
upstream rebase and is required separately):
- c512 -fa1 -mla3 indexer ON: 2.1983 (== prior baseline; build healthy).
- Long-ctx FA decode, 2735-tok recall prompt, -mla3 -fa1 temp0, reuse ON (default): coherent,
correct deep-context recall ("Dr. Mariana Velasquez ... Daniel Okonkwo") on BOTH the fixed and
the unfixed binary.
- ub128 PPL -fa1 -mla3 reuse ON, unfixed: 1.7239/1.8211/2.1888/2.4517, healthy (no inflation).
Honest scope: the bug is real in code but LATENT for GLM-DSA at its configured top_k=2048
(permissive selection keeps the genuinely-attended recent blocks even when reuse leaves some
recent index-key cells stale), unlike MSA's tighter top-k where it inflated PPL ~2x. The fix is
correct and prevents the latent corruption from biting at any tighter top_k / longer ctx /
future serving config. The pre-P2P-patch "nan" seen at ub128 was P2P corruption, not this bug.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-DSA: convert sparse-attention control from env vars to CLI args (off by default)
Implements ikawrakow's direction from discussion #2040: the DSA sparse
indexer must be controllable via command-line argument (not environment
variables), and must be OFF by default for now.
Control surface, before -> after:
DSA_INDEXER_DISABLE (env, inverted: on-by-default) -> --dsa / -dsa
(cparams.dsa, default false; opt-in, dense-by-default)
DSA_TOPK_OVERRIDE (env) -> --dsa-top-k N / -dsatk N
(cparams.dsa_top_k, default -1 == model's configured indexer_top_k)
DSA_HADAMARD_DISABLE, DSA_SINK (env) -> kept as DEBUG-ONLY env
knobs (clearly commented; no CLI surface, not system on/off controls)
Plumbing mirrors existing boolean/int feature flags (-mla, -khad):
include/llama.h llama_context_params {bool dsa; int dsa_top_k;}
src/llama.cpp default_params (false / -1); cparams assignment
src/llama-cparams.h llama_cparams {bool dsa=false; int dsa_top_k=-1;}
common/common.h gpt_params {bool dsa=false; int dsa_top_k=-1;}
common/common.cpp arg parse + help text + cparams copy
src/graphs/build_deepseek2.cpp gate now checks cparams.dsa instead of
getenv; top-k override reads cparams.dsa_top_k. Stays arch-gated to
LLM_ARCH_GLM_DSA. When --dsa is off (default) the indexer function is
never called -> existing dense MLA path, byte-identical to no-feature.
Validation (GLM-5.2-UD-IQ2_M, 3x P100, -ngl 99 --cpu-moe -mla 3 -fa 1,
wikitext-2, 4 chunks @ c2560):
--dsa OFF (default, dense): PPL 2.4151 (graph nodes 4166)
--dsa ON, default top_k=2048: PPL 2.4697 (graph nodes 8846)
--dsa ON, --dsa-top-k 1024: PPL 3.5107
Off-by-default runs the dense path; ON activates the indexer (node count
jumps, PPL shifts as the top-k mask bites once n_kv > top_k). No env var
is consulted for the primary on/off or the top-k knob.
Graph-parallel (-sm graph) interaction (the item ikawrakow flagged):
Under -sm graph the MLA layers are TP-split (wo->extra) and route to
build_deepseek2_tp_attention(), which contains NO indexer code. So --dsa
is silently a NO-OP under -sm graph: it does not error or crash, it runs
dense. Empirically, --dsa --dsa-top-k 1024 under -sm graph gives
PPL 2.4308 (chunks 1.6967/1.7906/2.1664/2.4308) -- the dense baseline
(2.4151), NOT the DSA top_k=1024 numbers (3.5107). The 0.016 delta is
f16 TP-reduce numerics, not DSA. Conclusion: DSA "works under deepseek2"
only on the non-TP (layer) path; serving DSA with -sm graph would require
wiring the indexer into the TP attention path (or a dedicated DSA arch).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-DSA: warn that --dsa is inactive under -sm graph/attn (TP path runs dense MLA)
The DSA lightning indexer is built only in the layer-mode (non-TP) attention
path. Under -sm graph / -sm attn the tensor-parallel attention path has no
indexer, so --dsa would silently run dense MLA. Emit a clear one-time
LLAMA_LOG_WARN at context creation instead of degrading silently.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-DSA: drop in-tree dev reference docs from the PR branch
DSA_REFERENCE.md and the R740 progress note are development scratch, not
part of the submission. Remove them so the PR diff is code-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* GLM-DSA: fix CPU-only crashes in the sparse-attention path
PR #2045 adds GLM-DSA sparse attention but was validated on CUDA (--cpu-moe).
A CPU-only build (-ngl 0 --dsa) crashes in four spots where the CUDA backend
tolerates something the CPU backend does not. These make GLM-5.2 --dsa run
coherently on CPU; with --dsa off they are no-ops (DSA CPU path only).
1. set_rows into an F32 dest segfaults (ggml.c set_rows_f32):
type_traits[F32].from_float is NULL, so the DSA sparse-mask scatter calls a
NULL fn (segfault at ip=0). memcpy when the dest is F32. CUDA has a real F32
set_rows path, so this only bit the CPU build.
2. ggml_add(F32 score, F16 mask) aborts on CPU (build_deepseek2_dsa_indexer and
build_deepseek2_dsa_sparse_mask): under -fa 1 the dense KQ_mask is F16 and CPU
add only accepts F32+F16 when src0 is F16. Cast the causal mask view to F32.
CUDA's add accepts the mixed types.
3. dsa_fa_mask dim-1 concat must be F32 on CPU (build_deepseek2_dsa_fa_mask):
CPU ggml_concat only supports F16 along dim 0; do the row (dim-1) concat in
F32 then cast the result to F16. CUDA supports the F16 dim-1 concat.
4. indexer k_norm epsilon is 0 -> ggml_norm aborts (llama-hparams.cpp): the
lightning-indexer k_norm is a non-RMS LayerNorm using f_norm_eps, but the
GLM-DSA GGUF only carries the RMS eps so f_norm_eps stays 0
(GGML_ASSERT(eps > 0)). Mirror the RMS eps. CUDA's norm doesn't assert on eps=0.
Validated: GLM-5.2 UD-Q4_K_M, single-socket Xeon w7-2475X, CPU-only (-ngl 0 --dsa)
- coherent at 49K+ ctx, correct 30K needle retrieval, prefill flat with length
(~32 tok/s, the O(L) DSA signature) vs the dense build's O(L^2) decline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* DSA: loop over attention heads + use builtin Hadamard
* DSA: ggml_blend
* DSA: remove a bunch of unnecessary ggml_cont
* DSA: fix CUDA blend - but something is still wrong
* DSA: use ggml_top_k instead of ggml_argsort when FA is ON
* CUDA: add CUB based argsort
* DSA: avoid graph leaves
* Various
* GLM-5.2 DSA: IndexShare (shared layers reuse full-layer top-k)
GLM-5.2's indexer_types marks 21 'full' layers that compute their own
lightning-indexer top-k and 57 'shared' layers that reuse the previous
full layer's top-k. This port computed an independent top-k on every
layer, which mis-selects keys on the 57 shared layers (the transformers
reference sets indexer=None on shared layers and reuses prev_topk).
Shared layers now reuse the most-recent full layer's selection. Full/
shared map derived from the config rule (full iff il<=1 or il%4==2),
which reproduces indexer_types exactly; loader can later override from
GGUF metadata. Built on #2063's tree; head-loop/ggml_hadamard/ggml_blend/
argsort/FA-mask unchanged.
4K PPL (unsloth IQ2_M, top_k 2048, CPU): DSA-on 3.1922 -> 2.7111, dense
2.6972 (~97% of the gap). top_k>=n_kv reproduces dense exactly. Single-
seq and 4x8 parallel decode coherent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Apply suggestion from @ikawrakow
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: mgkwill <168222+mgkwill@users.noreply.github.com>
Co-authored-by: Kawrakow <iwankawrakow@gmail.com>
* host-swap tensor loop
the host-swap functionality is only triggered when the certain env. variables are declared
* target_include_directories tweak
* hot-swap tensor support
two intrusions:
1.) at the model loading to collect the snapshot
2.) the modification of the `/health` HTTP endpoint to be able to trigger the hot-swap via sending the `llama-server` the HTTP-request.
*both a braced by the specific env. variables
* hot-swap tensor support; graph invalidation
ggml_backend_cuda_invalidate_graphs export
* hot-swap tensor support
graph invalidation implementation; extended debug output (commented out)
* llama_reload_changed_tensors export
* tensor hot-swap on-demand reload
cpu-only/hybrid/gpu-only with split mode layer/graph full support implementation
* docs
* reuse the gguf parsing from llama.cpp
gguf_init_from_file, gguf_find_tensor, ggml_get_tensor
* remove the manual scheduling for hybrid inference
* update docs
* tensor shape validation
* update docs
* update docs
accidentally wiped the previous changes; so recovered them
* revert the GGML_CUDA_MAX_DEVICES to 16
* update llama_reload_changed_tensor
update llama_reload_changed_tensor, revert CMakeLists.txt
* update llama_reload_changed_tensor
* GGML_MAX_SRC
GGML_MAX_SRC compile-time definition support
* GGML_MAX_SRC
GGML_MAX_SRC compile-time definition support
* GGML_MAX_SRC
GGML_MAX_SRC compile-time definition support
* llama_reload_changed_tensor
update llama_reload_changed_tensor definition
* refactory
move the tensor-reloading implementation to llama-reload.cpp, llama-reload-info.h; some bugfixes and code reduction
* revert
added back the missing newline
* update docs
* reload_info constructor
* bugfix: cpu-only
TODO: improve the working environment by compiling for multiple hardware configurations; possibly make a test pipeline
* cpu-only bugfix
set the fix again after unsuccessful sync with main
* windows os compilation fix
#include <string>
* fix windows os build
error C2039: 'string': is not a member of 'std'
* remove dead file
* implement perplexity in server
* Revert "implement perplexity in server"
* MLA tensor parallelism under -sm graph (DEEPSEEK2/GLM_DSA/MISTRAL4)
Extends -sm graph (split-mode graph) to MLA-style attention across the
DEEPSEEK2, GLM_DSA, and MISTRAL4 architectures. Previously these archs
fell back to -sm layer regardless of the user's flag.
Implementation:
- Per-rank attention build in build_deepseek2_tp_attention with
view-sliced FlashAttention, split-buffer output projection, and
ggml_reduce across devices
- wk_b / wv_b absorbed weights replicated per device via materialize()
in llm_prepare_mla (these can't live in a split buffer)
- KV cache replication path (replicated_k_l) for graph-mode TP
- distribute_mla_tensors_for_split_mode_graph routes attention/norm
tensors into ctx_split; expert tensors stay per-layer
- Implements ggml_backend_cuda_split_buffer_get_tensor for the
replicated / row-split / col-split inverse paths
- Early-reject guard in src/llama.cpp that auto-downgrades -sm graph
to -sm layer (with a warning) when incompatible loader flags are set:
-ncmoe, -cmoe, -ot, -rtr, -muge
New CLI flag:
- -gap | --graph-attn-precision <f16|f32> (default f16)
See the PR description for the full validation matrix (3 archs x 2/4/8
GPU counts), perf numbers, VRAM accounting, and known limitations.
* Some tweaks
* materialize lambda: per-head split for graph-mode tp_replicate
7dd19e19 changed wk_b/wv_b distribution from mirror to per-head split
(split_dim=2) via prepare_split_tensors. That path only fires when
wk_b/wv_b are loaded from GGUF.
Models that store only wkv_b in GGUF derive wk_b/wv_b at load via
llm_prepare_mla, going through the materialize lambda, which was
untouched and still produced mirror replicas (split_dim=-1, full n_head
per device).
build_deepseek2_tp_attention now does mul_mat(wk_b_local, q_nope_perm)
without the prior view_3d slice, so a mirror replica passes an n_head
tensor where the kernel expects n_head_local. Result: silent SIGSEGV
right after model load.
Mirror logic in materialize is replaced with the same per-head split as
prepare_split_tensors: head_offsets derived from wo split, each rank
gets a tensor with ne[2]=n_head_local, data copied from the appropriate
source byte slice. Singular `computed` tensor keeps full metadata for
tensors_by_name lookups.
Tested: 8x3090, -sm graph -mla 3 -fa on now boots cleanly and
sweep-benches without crash. Log confirms new path: "Computed
blk.X.attn_k_b.weight ... split across N devices on dim=2".
* cleanup: indent fix + remove dead view_3d slicing and debug printf
- build_deepseek2.cpp: re-indent the self_attention block in
build_deepseek2_layer_attention (lines 253-670). Block was at column 0
inside a function body; now at the expected 4/8-space indent.
- build_deepseek2.cpp: drop the commented-out view_3d slicing and debug
printfs left over after 7dd19e19's switch to direct mul_mat on
per-rank wk_b_local / wv_b_local. Update the stale 'wk_b is
replicated (split_dim=-1)' comment to match the new split_dim=2
reality.
- ggml-cuda.cu: remove the leftover debug printf in
ggml_backend_cuda_split_buffer_get_tensor.
No behavior change. Verified with a clean rebuild and DSV2.5 +
GLM-4.7-Flash sweep-bench runs.
* llm_load_tensors: gate incompatible-flag warning to MLA archs
The -ncmoe / -rtr / -muge / -ot warning under -sm graph currently fires
for all archs that support graph mode. That's an over-reach: the
incompatibility is specific to the MLA TP paths (DEEPSEEK2, GLM_DSA,
MISTRAL4) — Gemma4 graph mode existed pre-PR and works with those flags.
Gate the warning to MLA archs only.
Also refreshes two stale comments left over from the wk_b/wv_b
mirror -> per-head-split rewrite:
- src/llama.cpp llm_prepare_mla: "Replicate wk_b/wv_b ..." now reads
"Per-head split wk_b/wv_b ..." to match what the materialize lambda
actually does post-823a39e2.
- src/llama-load-tensors.cpp distribute_mla_tensors_for_split_mode_graph:
drop the wkv_b row-split mention (wkv_b is no longer created under
graph mode after 7dd19e19) and correct the wk_b/wv_b distribution
description (per-head split, not per-device replicated).
---------
Co-authored-by: Kawrakow <iwankawrakow@gmail.com>
ikawrakow asked the seq_rm zeroing be hoisted out of the per-cell loop and
preferably done as part of the compute graph. This rewrites the fix:
- llama_kv_cache gets a pending_recurrent_reset bitmap, sized to
qnext_state_slots and indexed by slot index.
- llama_kv_cache_seq_rm only marks the bitmap; no CPU-side tensor
writes, no work inside the per-cell loop.
- delta_net::build_layer_attn_linear ORs the bitmap into the existing
reset_state flag, so the in-graph ggml_scale(state, 0.0f) path that
already handled batch.pos[0] == 0 now also handles slot reset.
- build_qwen3next and build_qwen35 clear the bitmap after
ggml_build_forward_expand, once per graph build.
- llama_kv_cache_clear also drops the bitmap, since the full-buffer
clear it already does makes any pending in-graph reset redundant.
Previous behavior is preserved (every recurrent layer column for the
released slot is zeroed before the next request reuses it), but the work
moves from the CPU into the compute graph, runs once per graph instead
of once per cell, and reuses the existing reset path referenced in the
review.
* server: spec checkpoints for recurrent models
* fix: save/restore sampler state during speculative checkpoint
When speculative decoding rejects draft tokens and restores the
recurrent state checkpoint, the sampler (RNG, grammar, prev tokens)
must also be restored to maintain consistency. Without this, the
sampler state reflects the rejected draft tokens, leading to
potential divergence.
Uses common_sampler_clone() to snapshot the sampler before the
speculative batch decode, and restores it on rejection.
* server: snapshot recurrent state in tensor
* reset ngram mod state for rejected tokens
* server: refactor checkpoint state logic
* speculative: fix sampler for checkpoints
* recurrent model: implement recurrent kernel checkpoint
* recurrent model: refactor api
* spec: free rbudget before overwriting
* wip: separate llama_context for MTP with graph reuse
* wip: fix KV cache desync with separate MTP context
* refactor: remove dead mtp logic code, encapsulate KV mirroring
* mtp-context: derive args directly from the main model's context
* mtp: fix kv cache positions
* clean small comments
* minor refactor for context shift
* wip: port MTP architecture
Ports the Multi-Token Prediction (MTP) architecture to the older `llama.cpp` codebase used by `ikllama`.
Changes include:
- Updating `llama_batch` to support `mtp_params`.
- Modifying `llama_decode_internal` (and `encode`) to handle MTP operations (Warmup, Update, Draft).
- Adding public APIs for MTP state management (`llama_set_draft_input_hidden_state`).
- Adapting the embedding extraction logic to skip MTP update passes.
* Refactors `server_slot` to support generic speculative decoding (MTP or Draft Model).
* core: enable hybrid outputs (logits + embeddings) for MTP support
* fix(mtp): correct KV-cache slot finding for updates
* fix(mtp): persist hidden states to prevent context corruption during drafting
* refactor(mtp): clean unused code
* fix(mtp): update server to new functions name
* fix(mtp): fix graph and save hidden state
* mtp: refactor integration, context params and kv cache search
* mtp: fix hidden state extraction and speculative acceptance flow
* server: fix MTP warmup for long prompts and reset token buffer
* llama: refactor MTP operation state to context parameters
* server: fix n_past calculation in MTP acceptance
* llama: fix mtp enable flags
* speculative: refactor MTP to use common_speculative interface
* context: remove unused signatures
* clip: fix deprecated enum-enum conversion warning
* common: fix format string crash in help message
* context: fix mtp activation logic
* llamat: always use the extracted embedding
* llama: get all embeddings to kv cache
* llama: revert logit to not run mtp for not supported arch
* llama: allocate all the n_outputs for MTP
* wip
* server-context: get only the last embedding for hidden state
* ggml-backend: fix array of bounds in debug build
* server-context: run mt kv update to each prompt batch
* revert segmentation fault fixes
* glm-mtp(feat): optimize graph embedding and recursive drafting
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
Loads and starts running, crashes with illegal memory access in
quantize_mmq_q8_1. This almost always indicates NaNs in the input
to the MoE FFN part.
* WIP
* WIP
Loads and runs, wrong results (very high PPL)
Performance looks promising, around 25% better than previous sm graph.
Needs f32 or bf16 graph reduce type.
* WIP - still wrong
* Fix after rebase
* WIP
* WIP
* This seems to be working for dense Qwen3.5!!!
* WIP: Qwen3-Next is not quite working
* Some cleanup
* Disable Qwen3-Next for now
* Disable graph parallel when mmproj was specified
* Read/write split recurrent state
* That should not crash
* Re-enable vision - it works now
* Recurrent layers should now be counted for split cache
* wip: port MTP architecture
Ports the Multi-Token Prediction (MTP) architecture to the older `llama.cpp` codebase used by `ikllama`.
Changes include:
- Updating `llama_batch` to support `mtp_params`.
- Modifying `llama_decode_internal` (and `encode`) to handle MTP operations (Warmup, Update, Draft).
- Adding public APIs for MTP state management (`llama_set_draft_input_hidden_state`).
- Adapting the embedding extraction logic to skip MTP update passes.
* Refactors `server_slot` to support generic speculative decoding (MTP or Draft Model).
* core: enable hybrid outputs (logits + embeddings) for MTP support
* fix(mtp): correct KV-cache slot finding for updates
* fix(mtp): persist hidden states to prevent context corruption during drafting
* refactor(mtp): clean unused code
* fix(mtp): update server to new functions name
* fix(mtp): fix graph and save hidden state
* mtp: refactor integration, context params and kv cache search
* mtp: fix hidden state extraction and speculative acceptance flow
* server: fix MTP warmup for long prompts and reset token buffer
* llama: refactor MTP operation state to context parameters
* server: fix n_past calculation in MTP acceptance
* llama: fix mtp enable flags
* speculative: refactor MTP to use common_speculative interface
* context: remove unused signatures
* clip: fix deprecated enum-enum conversion warning
* common: fix format string crash in help message
* context: fix mtp activation logic
* qwen3next: add architecture support and recurrent-state fixes
* qwen3next: optimize broadcast sub and single-seq ssm conv
* cuda: build MoE row mapping on device in mul_mat_id
* cuda: add guarded multi-seq fast path for ssm_conv
* docs: update qwen3next perf report for cuda MoE/SSM tuning
* cuda: reduce qwen3next moe/ssm sync overhead and refresh eval
* qwen3next: split cpu/cuda eval builds and tune PP scheduling
* qwen3next: harden seq-state flow and support optional dense FFN layers
* qwen3next: trim delta-net graph overhead in chunking path
* qwen3next: remove redundant v_conv cont in delta path
* qwen3next: avoid extra cont on linear attention output
* qwen3next: drop redundant cont before recurrent state flatten
* qwen3next: keep recurrent state in 4d layout through delta path
* qwen3next: add fused delta-net op and wire model path
* tests: add backend-op coverage for ggml_delta_net
* qwen3next: add runtime switch for fused delta-net path
* docs: refresh qwen3next perf review and benchmark matrix
* qwen3next: default fused delta-net off and document quality checks
* qwen3next: add decode-only fused delta mode
* qwen3next: make fused delta safe by default and fix fused tensor layout
* qwen3next: warn when forcing fused decode mode
* qwen3next: add fused-delta regression runner script
* qwen3next: integrate fused regression into eval harness
* qwen3next: clean up chunked delta-net shape handling
* qwen3next: add absolute sanity guards to fused regression
* qwen3next: add unified regression runner script
* qwen3next: disable flash-attn for cpu-only contexts
* docs: reconcile qwen3next status and remaining upstream gaps
* common: add qwen3next fused-delta runtime flag
* cuda: add qwen3next delta-net kernel dispatch override
* docs: update qwen3next quality and serving baseline findings
* qwen3next: keep fused delta on safe path and remove PR artifacts
* qwen3next: align autoregressive delta-net decode layout
* Revert "qwen3next: align autoregressive delta-net decode layout"
This reverts commit 9241164a5ea9e032a2456fbf2dd0bf798b264fd7.
* cuda: port solve-tri fast-paths for qwen3next delta-net
* qwen3next: add fused-delta runtime flag and drop env toggle
* qwen3next: make fused delta single-flag and default on
* Account for GPU arch differences
* Revert "cuda: build MoE row mapping on device in mul_mat_id"
This reverts commit 89e9ecfa840b04e88699ab3803eb732cd78727f9.
* qwen3next: drop non-essential MoE scheduling and split heuristics
* qwen3next: avoid generic ggml_sub broadcast changes
* llama: restore only_active_experts log message
* Remove unnecessary hacks, disable fusion for now.
* qwen3next: port hybrid recurrent state memory semantics
* qwen3next: clean up recurrent state slot plumbing
* qwen3next: fix hybrid V-cache layout plumbing
* qwen3next: guard recurrent state slots against kv capacity
* qwen3next: persist recurrent state in session data
- serialize/restore qwen3next cache.s_l in state/session paths\n- bump session and sequence-state file versions for format change\n- fallback to single-token chunking for mixed repeated seq_id batches
* qwen3next: drop unused fused-delta builder path
- remove dead build_delta_net_fused lambda\n- remove unused llm_build_context::fused_delta member
* qwen3next: remove unused fused-delta CLI/context plumbing
- drop -fd/-no-fd options and related YAML dump field\n- remove fused_delta fields from public/internal context params\n- remove fused_delta assignment and logging in context init
* ggml: remove unused DELTA_NET operator stack
* Missing include
* Reorder ops/unary ops
So we don't change again the enum values of the mul mat ops
* Minor
* Discard unnecessary changes in llama-build-context.cpp
* Minor
* Revert "Discard unnecessary changes in llama-build-context.cpp"
This reverts commit edadb80ed68c4c0831e9c22609a9a3af19be9735.
* Increase GGML_SCHED_MAX_SPLITS - required for larger u-batches
* Fix CPU concat in the TG case: 7.25 -> 10.5 t/s for Qwen3Next
* Fix CPU sum_rows: 10.5 -> 13.6 t/s for Qwen3Next
It was single-threaded and was taking ~25% of the computation time
during TG. It is now down to 2%.
Strangely enough, I measure 13.6 t/s with llama-bench, but if I
let the model give me an actual response with llama-cli, I get close
to 17 t/s.
* Fix CPU scale: 13.6 -> 16.7 t/s for Qwen3Next
For Qwen3Next there is a scale op on a largish tensor (548k elements)
that has a single row for TG, so was done in a single thread.
We now simply use blocks of 1024 elements.
* Optimize CPU mul: 16.7 -> 17.6 t/s for Qwen3Next
* CPU: fuse transpose -> cont -> sum_rows -> transpos: 17.6 -> 23.1 t/s for Qwen3Next
* Optimize CPU repeat: 176 -> 200 t/s for Qwen3Next PP-512
* Multithreading for OP_SUB
* Don't commit with timing trace on
* Multithread neg and sigmoid
* Be able to turn on/off fusion more easily (CPU)
* Name the mul_mat ops so we know where the time goes
* WIP
* Much better PP on CUDA
* CUDA: fuse transpose -> cont -> sum_rows -> transpose
Needs non-coontiguous variant of sum_rows.
On the CPU this gave 30+% improvement in TG performance,
on CUDA ist is disapointing 6-7%. I guess, this is because
Georgi's cont CPU implementation was so bad that skipping
it made such a big difference.
* CUDA: faster mul for special case relevant for Qwen3Next
Worth 1% in TG
* Fix CPU OP_CONT
---------
Co-authored-by: yurko <yurko@local>
Co-authored-by: Yurko <yurko@example.com>
Co-authored-by: yurko <yurko@pop-os.tail5a1a6b.ts.net>
Co-authored-by: Yurko Hoshko <YurkoHoshko@users.noreply.github.com>
* Remove most of split mode row
* WIP
* WIP: also allocate the KV cache using tensor split
* WIP: it runs with wrong result
But it also looks like the backend scheduler is not going to help:
* It copies mask and input positions to GPU 0
* => RoPE ops must run on GPU 0
* => To proceed attn evaluation, GPU 1 must wait for GPU 0 to finish its
entire attn calculation
* Same with FFN. The rms_norm gets scheduled on GPU 0. Hence, GPU 1 must
wait for GPU 0 to finish its entore FFN calculation before it can
start (as it needs to copy the result of rms_norm from GPU 0)
* => Seems useless without writing a bespoke TP scheduling
* WIP
* This works, but it is slow
* This is slightly better
the graph is still not being computed in parallel.
Why? Because the scheduler creates graph splits where the
result of the computation on one GPU becomes an input for the
other split. Hence, to trigger the computation on the second GPU
one needs to wait for the computation on the first GPU to finish,
even thiough the two can be done in parallel up to the sunchronization
point. So, all that is left to do is to trick the scheduler to create
to splits that can be done in parallel, and then have a graph split
where the results get combined.
* Playing games with the scheduler
This change tricks it into doing the right thing^TM.
Still quite a bit slower than split mode layer for the 8B LlaMA model.
But for the 70B LlaMA it now beats split mode layer for TG:
28 t/s vs 24.4 t/s. PP is 627 t/s vs 744 t/s.
In comparison, split mode "row" in mainline gets
484 t/s PP and 19.3 t/s TG.
* Fix attn split
Granularity for Wq, Wo is not just head size, but
head size * gqa_ratio.
Else the Wk, Wv tensors end up not being a multiple of the
head size when we divide the split determined by Wo with
the gqa_ratio.
* Show memory used per device
* Make it work with partial offload
but no tensor overrides yet, just ngl < num_layers.
* Allow for f16 source in fused_rms_norm
* This results in faster PP.
Now PP is faster than split mode layer for L3-70B.
* Rename split mode "row" to split mode "graph"
* Leave FFN partial results as f16
* WIP GLM4.5 - runs with wrong results
* WIP GLM4.5 - this works
PP is already better than split mode layer, but TG for zero context
is kind of low - 60 vs 92 t/s. TG becomes better than split mode layer
at around 20k tokens. PP at 26k tokens is 1.55X of sm layer.
* Work around compiler bug
It issues a warning that there is an extra semicolon outside of a function,
but there isn't. If I remove the anonymous namespace and turn the
functions inside into static, the warning disapears, so clearly
a compiler bug.
* Make graph reuse work with split mode graph
* Remove more split mode row remnants
* WIP tensor overrides
Runs with wrong results, don't see where the issue could be.
* This works but is slow
Still does not work for row-interleaved quants
* Slightly better
* Slightly better
* Row-interleaved quants work
* Better
* Minor
* Guarad against using split mode "graph" for unsupported models
* Guards against using merge_qkv with split mode "graph"
* WIP split mode attn
Works for LlaMA models, but not for GLM-4.5.
Doesn't seem to improve performance, so I guess no point in trying to
fix it.
* Split mode graph for qwen3moe
* Try to better distribute the splits
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
* Add mainline compatible FA command line option
* Graph reuse: add command line argument to turn it on
* WIP
* This seems to work
* This is perhaps cleaner
* Change the command line option to -gr
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
* llama_model and llama_hparams
* llama_build_context
Surprisingly small reduction in llama.cpp compile time given
the reduction in LOCs (22k -> 14k)
* LLM_TN
llama.cpp compilation: 50 s -> 33 s
* llama_quantize
* arch names
* All graph building is now in llm-build-context.cpp
* hparams loading
llama.cpp is now just 9300 LOC, but still takes 32 seconds to compile.
* We are now at 6 seconds to build the src folder
* load -> create
We are not actually loading the tensors, but just creating them.
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Kawrakow
Joel Farthing
mb8565
magikRUKKOLA
SamuelOliveirads
Samuel Oliveira Alves
David Young
markaalonzo
firecoperana
firecoperana