* Update parameters.md
- Add new parameters
- Update modified parameters
- Add graph parallel new arch
- Fix some words case
* Update README.md
- Update supported models list
- Add the new features
* model: Ling-3.0 (bailingmoe3) runtime support
* model: Ling-3.0-tiny support
---------
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
The draft block was seeded at last_target_pos (the newest committed
feature row = id_last's predecessor), placing the whole block one
position early vs mainline's [id_last @ n_past, ...] convention and
colliding the seed with the newest cross-KV row. Shift the common-side
batch and the graph-side SWA mask base coherently.
* Adding Muse-Glimmer support
* Different rms_eps for post norm ops
* Need attn_post_norm split for Muse-Flimmer
* WIP: split mode graph
* Forgot this file
* Clean it up
* Minor
* Muse-glimmer: Slightly better split mode graph (+2% TG)
build_llama passes nullptr for the rope_factors_in argument, so
rope_freqs.weight never reaches ggml_rope_ext and llama3 rope frequency
scaling is not applied. The LLAMA_SPLIT_MODE_GRAPH branch inside
build_std_attention falls back to model.layers[il].rope_freqs, so only
the standard path is affected.
The DFLASH case was inserted into the NEOX fall-through group in
llama_rope_type(), so the 39 cases above it now return LLAMA_ROPE_TYPE_NORM
instead of LLAMA_ROPE_TYPE_NEOX.
Move DFLASH into the NORM group so it keeps its intended rope type and the
others fall through to NEOX again.
* CUDA indexer topk: this is better for PP
* Don't overstep
* Cleanup
* Allow Q8_0 cache in the CUDA DSA implementation
* DS4: do not cast caches to f32
* Fix massive inefficiency in CUDA Q->f32/f16 and f32/f16->Q copies
* Re-enable -ictk | --indexer-cache-type-k
* CUDA indexer topk: this is better for PP
* Don't overstep
* Cleanup
* Allow Q8_0 cache in the CUDA DSA implementation
* DS4: do not cast caches to f32
Compute it before the reuse check, pass it to can_reuse_graph(), and reuse the same
value when the graph is rebuilt, so it is computed exactly once whether the existing
graph is kept or rebuilt.
Verified at np=6, same binary, only IK_LEGACY_GRAPH_REUSE differing: legacy 6/12
replies kept their own codeword, this branch 12/12. The value at the call site matched
the value at the store site on 50 of 50 rebuilds sampled at np=2 and np=6.
* Fix wrong output for hybrid/recurrent models at -np > 1 (graph reuse key)
Hybrid and recurrent architectures return silently wrong output when more than
one sequence is resident. No crash and no warning: every slot keeps producing
fluent text, it is just no longer conditioned on that slot's prompt, and slots
degenerate into repetition loops. Reported in #1932.
Three things have to line up, and on these architectures they do:
1. can_reuse_graph() keys reuse on the ubatch SHAPE. Two consecutive decode
steps for different sequences match on every field it checks.
2. update_cache_copies() re-points the baked view_offs for K/V, but only for
attention layers -- it skips recurrent ones via
is_attn_layer = !hparams.is_recurrent(il).
3. The delta-net bakes the recurrent state row into the graph as a
compile-time view offset, not as an input tensor.
So a graph built for sequence A is reused to decode sequence B and nothing
re-points the recurrent state: every sequence reads and writes sequence A's
state. On Qwen3.6-35B-A3B only 10 of 40 layers carry a KV cache, so the 30
layers that silently share state are three quarters of the network.
Fix: extend the reuse key with a fingerprint of the ubatch's sequence
composition -- which sequences, in what order, and which start at position 0.
That last term matters because a state reset is baked into the graph as a node;
it mirrors exactly the condition the builder itself uses (batch.pos[i] == 0 in
build_layer_attn_linear). Gated on llm_arch_is_hybrid() ||
llm_arch_is_recurrent() on both the compare and the compute side, so
architectures that never consult the fingerprint do not pay to build it.
Evidence, RTX 3090 / sm_86, Qwen3.6-35B-A3B-UD-IQ4_XS, 6 concurrent requests
each carrying a unique codeword, 600-token generations, two rounds:
before (IK_LEGACY_GRAPH_REUSE=1): 7/12 replies degenerate
after: 0/12
np=1 is unaffected. Throughput is 132.9 tok/s against 132.4 before, and the
fingerprint is a pure additional invalidation -- it can only ever add reuse
misses, never remove them -- so counting the misses it causes on its own bounds
its cost exactly. Over 13000 can_reuse_graph() calls at np=1:
calls=13000 hit=12949 miss_other=51 miss_fingerprint_only=0
Zero, so np=1 graph reuse is bit-identical to before this patch. That covers
MTP, which runs at n_parallel == 1: its draft/verify ubatch alternation was
already keyed by the existing n_tokens / mtp_op_type / mtp_step_idx /
mtp_n_heads checks, and all the fingerprint adds beyond those is per-token
seq_ids and the pos == 0 flags, both constant at np=1 during decode. prev and
prev_mtp are populated through the same reference binding, so the MTP cache
carries the fingerprint too.
IK_LEGACY_GRAPH_REUSE restores the previous behaviour so the before/after above
can be reproduced from a single build.
* Address review: fold the arch and legacy checks into the fingerprint
llama_graph_bakes_seq_state moves above the fingerprint; the fingerprint takes the arch
and returns 0 for architectures that do not bake sequence state, and when
IK_LEGACY_GRAPH_REUSE is set. Both call sites become a plain call and the two guards
live in one place.
Returning 0 keeps behaviour identical for everything else: stored and computed values
are both 0, so the comparison always matches and reuse proceeds as before this patch.
Re-verified at np=6, same binary, only the env var differing:
legacy 6/12 sequences degenerated -> DIRTY
fixed 0/12 -> clean
* openpangu: opt-in compacted sliding-window KV cache (--swa-compress)
* openpangu: shrink the compacted window and drop the zero fill
* openpangu: correct the --swa-compress state I/O refusal message
---------
Co-authored-by: Joel Farthing <262452229+joelfarthing@users.noreply.github.com>
Samuel Oliveira Alves
Kawrakow
Joel Farthing
Marian M.
rumas77
mb8565
abc-nix
ShubhamPriyadarshi