From 5d028a785543abf259dc6cd0fa4097c9cab683c3 Mon Sep 17 00:00:00 2001 From: Marvin Date: Sun, 6 Sep 2026 12:27:45 -0300 Subject: [PATCH] docs: log Ampere+Zen3 optimizations and add porting guide --- PORTING-OPTIMIZATIONS.md | 113 +++++++++++++++++++++++++++++++++++++++ README.md | 34 ++++++++++++ 2 files changed, 147 insertions(+) create mode 100644 PORTING-OPTIMIZATIONS.md diff --git a/PORTING-OPTIMIZATIONS.md b/PORTING-OPTIMIZATIONS.md new file mode 100644 index 00000000..3c43bd58 --- /dev/null +++ b/PORTING-OPTIMIZATIONS.md @@ -0,0 +1,113 @@ +# Agent porting guide: re-applying the Ampere+Zen3 optimizations + +**Use case:** upstream `ik_llama.cpp` has moved (new model arch, refactored file, new kernels) and you must re-apply this branch's optimizations to the new tree. Work top-down: OPT-01 → OPT-07. Each item is self-contained: *locate* (exact anchor), *transform* (exact edit), *verify*. + +**Target rig (do not re-tune for other hardware):** RTX 3090 24GB + RTX 3070 8GB (`sm_86`, no P2P) + Ryzen 5900XT 16C/32T (`znver3`, DDR4-3600). PP-bound long-context agentic sessions. Glove-fit non-portable builds are acceptable and expected. + +## Ground rules + +1. **Arch-gate every model-specific change.** Gate on `model.arch == LLM_ARCH_` (loader) or put the code in `src/graphs/build_.cpp` (graph). Never change shared paths unconditionally. +2. **New CUDA kernels are opt-in env flags, default off**, until A/B proves a win. Pattern: `getenv("DELTA_WY_CUDA")` → branch → exact fallback. See OPT-06. +3. **One optimization = one commit**, message prefix: `qwen4exp:`, `cuda:`, `tests:`, `ggml:`. +4. **Validate after every item** (see "Validation protocol"). If the anchor is gone (upstream refactor), locate the successor by searching the anchor's keywords, adapt the transform, and note the new anchor in the commit message. +5. **Reference implementation:** commits `e2728c85` (CPU/graph), `a47a8a65`/`f5b494b0`/`3a7d1016` (harness + WY candidate), `a34feeb0` (CUDA kernel), `c1a36daa` (harness CUDA fix), `7930df51` (MoE microbench). Use `git show -- ` to see the exact original diff. + +## OPT-01 — Merged up/gate expert projections (loader) + +- **Goal:** one `[2*n_ff, n_embd]` GEMM per expert batch instead of two; halves expert weight-streaming traffic on CUDA `mmq_id` and CPU IQK paths. +- **Locate:** `src/llama-load-tensors.cpp`, functions `create_std_ffn_exps` and `create_std_ffn_exps_from_meta`. Anchor string: `ml.merge_up_gate_exps && merge_up_gate_exps(tn, i, 0)` (appears in both functions, in the `else` branch where no fused `ug_meta` tensor exists in the file). +- **Transform:** in both functions, replace the anchor condition with: + ```cpp + const bool want_merge = ml.merge_up_gate_exps || model.arch == LLM_ARCH_; + merged = flags == 0 && want_merge && merge_up_gate_exps(tn, i, 0); + ``` + substituting the new arch enum. `merge_up_gate_exps()` already falls back gracefully when types/shapes differ — do not bypass it, only widen the opt-in. +- **New arch checklist:** if upstream's new arch stores experts as separate up/gate tensors (check its `create_*_tensors`), add its enum here. If the file already ships a fused up-gate tensor, this item is a no-op. +- **Verify:** model loads, `llama-bench -p 512 -n 0` PP within noise of unpatched, no `ggml_assert` on expert shapes. + +## OPT-02 — Fused PLE / convolution tap accumulation (graph) + +- **Goal:** remove 2 transposes + 2 conts per tap per layer in persistent-memory convolution loops. +- **Locate:** `src/graphs/build_.cpp`, the PLE conv helper (here `qwen4exp_ple_conv`). Anchor pattern: `ggml_reshape_1d(...)` on the tap weight followed by `ggml_mul(ctx0, ggml_cont(ctx0, ggml_transpose(ctx0, shifted)), wk)` and a re-transpose. +- **Transform:** + ```cpp + wk = ggml_reshape_2d(ctx0, wk, 1, hc_dim); // [1, hc_dim] broadcasts over rows + ... + ggml_tensor * term = ggml_mul(ctx0, shifted, wk); // [n_tokens, hc_dim], no transposes + ``` + Keep the existing `ggml_cast(..., GGML_TYPE_F32)` on `wk`. Math is identical (row-vector broadcast); confirm output shapes in comments. +- **New arch checklist:** any new arch with a tap/conv accumulation loop over a persistent cache gets the same treatment in its `build_.cpp`. +- **Verify:** perplexity spot-check (`llama-perplexity`, 1–2 shards of wikitext) matches unpatched to ≤0.001; PP bench neutral-to-positive. + +## OPT-03 — `k_x_step` 64 → 32 (CPU IQK MoE tiling) + +- **Goal:** Zen3 CCX locality for narrow expert GEMV; 64-wide reuse set thrashes the 2×32MB L3 on 5900XT. +- **Locate:** `ggml/src/iqk/iqk_mul_mat.cpp`, struct `MulMat`, **two** sites (`mul_mat_NxM` and the unary-op variant). Anchor comment: `This works best on my Ryzen-7950X`. Both are in the `#else` (non-aarch64) branch of `constexpr int k_x_step = 64;`. +- **Transform:** change both `64` → `32`, keep the `#ifdef __aarch64__` branch untouched. Carry a comment: `32 for Zen3 CCX locality; Zen4/Intel diff is small`. +- **Verify:** `test-moe-perf` (OPT-07) TG/PP neutral-or-better on this rig; full model TG must not regress >1%. + +## OPT-04 — Expert batching chunk `/32` → `/64` (ggml thread scheduling) + +- **Goal:** halve atomics/barriers on 32-thread Zen3 for narrow experts. +- **Locate:** `ggml/src/ggml.c`, `ggml_compute_forward_mul_mat_id` (anchor `ne01 / 32`) and `ggml_compute_forward_mul_mat_id_up_gate` (anchor `nr0_base / 32`). Full lines: `MAX(1, MIN(nth, (int)(ne01 / 32)))` and the `_ug` equivalent. +- **Transform:** `/ 32` → `/ 64` in both lines. The `MIN(nth, ...)` cap already protects TG load balance — do not touch it. +- **Verify:** TG-heavy bench (`-p 128 -n 128`) must not regress; PP should improve or hold. + +## OPT-05 — IQ4_XS AVX2 software prefetch (Zen3 DDR4 latency hiding) + +- **Goal:** hide DDR4 latency on the 4-bit expert GEMM weight stream during TG. +- **Locate:** `ggml/src/iqk/iqk_gemm_kquants.cpp`, function `mul_mat_qX_K_q8_K_T`, the `for (int i = 0; i < nb; ++i)` block-body loop. Anchor: `auto all_scales = deq.new_block(i, q8, accd);`. +- **Transform:** immediately before the anchor line, insert: + ```cpp + if (i + 4 < nb) { + __builtin_prefetch(deq.x + i + 4, 0, 3); + } + ``` + Weight stream is ~270 B/block; 4 blocks ≈ 1 KB stays in the L1 streamer window. If upstream renames `deq.x`/`new_block`, adapt to the new dequant-struct member holding the next weight block. +- **Verify:** TG tok/s neutral-or-better on IQ4_XS expert layers; remove if negative (single-line revert). + +## OPT-06 — Chunked WY delta-net (recurrent-state prefill) + CUDA kernel + +- **Goal:** chunked formulation of the GDN recurrent update with per-chunk `exp(diff)` rescaling; CUDA fast path for long prefills on hybrid archs. +- **Locate (CPU/reference):** `ggml/src/ggml.c` recurrent/delta-net forward op used by the arch. **Locate (CUDA):** `ggml/src/ggml-cuda/delta-net.cu`, dispatch function containing the sequential path; anchor: `saved_states == nullptr` guard region. +- **Transform:** + 1. Port `tests/test-delta-chunk.cpp` first (OPT-07) — it encodes the validated math (chunk sizes 64/32, exp-diff ratios, empty/oversized-chunk edge cases). The candidate must match the sequential reference bit-close on all 42 cases before any kernel work. + 2. Re-apply `delta_net_chunked_f32` in `delta-net.cu` (see `git show a34feeb0`): shared-memory tiled kernel, tiers for head-dim 64/128 + sequential fallback, chunk constants `DELTA_WY_CHUNK 64` / `DELTA_WY_CHUNK_SMALL 32`. + 3. Dispatch pattern (opt-in, default off): + ```cpp + const char * wy_env = getenv("DELTA_WY_CUDA"); + bool wy_wanted = wy_env && wy_env[0] == '1'; + if (wy_wanted && n_tokens > DELTA_WY_CHUNK && saved_states == nullptr) { + int chunk = (head_dim <= 64) ? DELTA_WY_CHUNK : DELTA_WY_CHUNK_SMALL; + ... chunked path ... + } else { ... existing sequential path, untouched ... } + ``` +- **Verify:** harness 42/42 on CPU **and** `--cuda`; then server A/B (`DELTA_WY_CUDA=1` vs unset) on ≥25k prefill. Promote to default-on only with ≥2% repeatable PP win; otherwise keep as opt-in infrastructure (current status: parity, default off). + +## OPT-07 — Test harnesses (port these, don't skip) + +- **`tests/test-delta-chunk.cpp`** (718 lines): correctness sweep (chunks × head-dims × dtypes × edge cases) + `--cuda` backend mode + chunk-size perf sweep. CUDA backend path **must** use a no-alloc ggml context (`c1a36daa` — asserts otherwise). Wire into `tests/CMakeLists.txt` following the existing `test-` target pattern (see `7930df51` for the minimal 9-line addition). +- **`tests/test-moe-perf.cpp`** (140 lines): MoE GEMM microbench for A/B without loading a 90GB+ model. Build-only target; run before/after OPT-03/04/05. +- **Rule:** any new arch or kernel gets harness coverage before server runs. Harness green (42/42 CPU + CUDA) is the merge gate. + +## Validation protocol (every item) + +1. **Build (glove-fit):** + ``` + cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON \ + -DCMAKE_CUDA_ARCHITECTURES="86-real" \ + -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_IQK_MUL_MAT=ON \ + -DCMAKE_CXX_FLAGS="-march=znver3" + cmake --build build --config Release -j$(nproc) + ``` +2. **Harnesses:** `build/bin/test-delta-chunk` → `PASS 42/42`; `build/bin/test-delta-chunk --cuda` → `PASS 42/42`. +3. **Micro:** `build/bin/test-moe-perf` before/after (≥3 runs, take median). +4. **Server A/B:** `llama-bench -m -p 4096 -n 0 -ts 4/1` for PP, `-p 128 -n 128` for TG; then the real 25k session flags. Accept: PP +2% or TG no-regression; revert anything else. +5. **Correctness:** `llama-perplexity` spot-check ≤0.001 drift after any graph/loader change. + +## Known non-goals (evaluated, deferred — do not redo without new evidence) + +- **QSA sparse gather kernel:** needs a new gather kernel; expected gain <0.5% at real `n_kv`. Revisit only with profiler data showing QSA >5% of PP time. +- **HC fusion:** ~0.1% of PP. Skip. +- **`--fit` + `--n-cpu-moe`:** mutually exclusive in `src/llama.cpp` (explicit error). Do not "fix" — use `--no-mmap` + manual placement. +- **Bench `-ts` syntax:** `4/1` (slash) for multi-GPU in `llama-bench`; comma means separate runs. Server uses comma. Do not "unify". diff --git a/README.md b/README.md index d5514bb7..cc1ea142 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,40 @@ [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT) +## Ampere + Zen3 optimization branch + +Target rig: RTX 3090 24GB + RTX 3070 8GB (`sm_86`, no P2P) + Ryzen 5900XT (`znver3`), running hybrid MoE models (`qwen4exp` / `qwen35moe` — Qwen3-Next, Qwen3.5-MoE, Qwen3.8-Flash-Next) with long (25k+) prompt-processing-bound agentic sessions. All changes are opt-in or arch-gated; default behavior on other hardware is unchanged. + +### CPU / graph optimizations +- **Merged up+gate projection for `QWEN4EXP` MoE** (`src/llama-load-tensors.cpp`): experts' `ffn_up_exps`/`ffn_gate_exps` are concatenated at load into a single `[2*n_ff, n_embd]` tensor per expert batch, turning two GEMMs into one and halving weight-streaming traffic on the CPU expert path. +- **Fused PLE convolution + tap broadcast** (`src/graphs/build_qwen4exp.cpp`): the per-tap `mul` + `add` sequence over the persistent-layer cache is replaced by a single broadcast-add over the stacked taps, cutting graph-node overhead per layer. +- **`k_x_step` 64 → 32 on the CPU MoE path** (`ggml/src/iqk/iqk_mul_mat.cpp`): smaller micro-panel for the AVX2 GEMV loop, fits Zen3 L1/L2 better for narrow expert activations. +- **Expert batching chunk `/32` → `/64`** (`ggml/src/ggml.c`): larger expert shards per thread, fewer barriers across 32 threads on the 16-core Zen3. +- **IQ4_XS AVX2 prefetch tuning** (`ggml/src/iqk/iqk_gemm_kquants.cpp`): software prefetch distances retuned for Zen3's prefetcher on the 4-bit expert GEMM hot loop. + +### DeltaNet (GDN) chunked prefill +- **Chunked WY representation** (`ggml/src/ggml.c`, `ggml/src/ggml-cuda/delta-net.cu`): alternative chunked formulation of the recurrent-state update with per-chunk `exp(diff)` rescaling, numerically validated bit-close against the sequential reference by `tests/test-delta-chunk.cpp` (42/42 cases, CPU and `--cuda` backend). +- **CUDA kernel `delta_net_chunked_f32`** (`ggml/src/ggml-cuda/delta-net.cu`): shared-memory tiled implementation (tiers for head-dim 64/128/sequential fallback), exact fallback outside the fast path. Opt-in via `DELTA_WY_CUDA=1`; default off after A/B showed parity (~193 vs ~195 tok/s prefill) on RTX 3090 — kept as infrastructure for future chunk sizes / head dims. +- **`--cuda` backend wiring for the delta-net harness**, so the chunked path is testable on GPU without a full server run. + +### Test harnesses +- **`tests/test-delta-chunk.cpp`**: correctness sweep (chunk sizes × head dims × dtypes × edge cases incl. empty/oversized chunks) for the WY candidate vs sequential reference; also serves as the GPU perf probe. +- **`tests/test-moe-perf.cpp`**: MoE GEMM microbenchmark for A/B of expert-path changes without loading a 94GB model. + +### Measured results (RTX 3090, Qwen3.8-Flash-Next UD-IQ4_XS, PP-bound) +- CPU/graph pass: ~195 vs ~190 tok/s prefill (+2.6%), TG neutral. +- Chunked-WY CUDA vs sequential: parity — merged, default-off. +- QSA sparse-gather and HC fusion evaluated and deferred (needs a new kernel; <0.5% expected gain at real KV lengths). + +### Recommended build for this rig +``` +cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON \ + -DCMAKE_CUDA_ARCHITECTURES="86-real" \ + -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_IQK_MUL_MAT=ON \ + -DGGML_CPU_ALL_VARIANTS=OFF -DGGML_CPU_ARM_ARCH=OFF +``` +plus `-DCMAKE_CXX_FLAGS="-march=znver3"` / CUDA `-gencode arch=compute_86,code=sm_86` for a glove-fit non-portable binary. See `docs/build.md` for details. + ## TL;DR This repository started as a fork of [llama.cpp](https://github.com/ggerganov/llama.cpp) in June of 2024 and was last synced with upstream in August of 2024. Compared to mainline `llama.cpp`, it offers additional SOTA quantization types and, in many cases, better performance. Various features related to LLM inference appeared here first before becoming available in llama.cpp. MLA, quant repacking, fused delta-net (known in `llama.cpp as "Gated Delta Net" - GDN), tensor parallel, MTP, DFlash, to just name a few.