docs: OPT-08 Bailing rescue+strip porting entry with rebase anchors

This commit is contained in:
Marvin 2026-09-07 22:28:10 -03:00
parent f9ffb6fb2d
commit 617732791e
1 changed files with 14 additions and 2 deletions

View File

@ -1,6 +1,6 @@
# Agent porting guide: re-applying the Ampere+Zen3 optimizations
**Use case:** upstream `ik_llama.cpp` has moved (new model arch, refactored file, new kernels) and you must re-apply this branch's optimizations to the new tree. Work top-down: OPT-01 → OPT-07. Each item is self-contained: *locate* (exact anchor), *transform* (exact edit), *verify*.
**Use case:** upstream `ik_llama.cpp` has moved (new model arch, refactored file, new kernels) and you must re-apply this branch's optimizations to the new tree. Work top-down: OPT-01 → OPT-08. Each item is self-contained: *locate* (exact anchor), *transform* (exact edit), *verify*.
**Target rig (do not re-tune for other hardware):** RTX 3090 24GB + RTX 3070 8GB (`sm_86`, no P2P) + Ryzen 5900XT 16C/32T (`znver3`, DDR4-3600). PP-bound long-context agentic sessions. Glove-fit non-portable builds are acceptable and expected.
@ -10,7 +10,7 @@
2. **New CUDA kernels are opt-in env flags, default off**, until A/B proves a win. Pattern: `getenv("DELTA_WY_CUDA")` → branch → exact fallback. See OPT-06.
3. **One optimization = one commit**, message prefix: `qwen4exp:`, `cuda:`, `tests:`, `ggml:`.
4. **Validate after every item** (see "Validation protocol"). If the anchor is gone (upstream refactor), locate the successor by searching the anchor's keywords, adapt the transform, and note the new anchor in the commit message.
5. **Reference implementation:** commits `e2728c85` (CPU/graph), `a47a8a65`/`f5b494b0`/`3a7d1016` (harness + WY candidate), `a34feeb0` (CUDA kernel), `c1a36daa` (harness CUDA fix), `7930df51` (MoE microbench). Use `git show <sha> -- <file>` to see the exact original diff.
5. **Reference implementation:** commits `e2728c85` (CPU/graph), `a47a8a65`/`f5b494b0`/`3a7d1016` (harness + WY candidate), `a34feeb0` (CUDA kernel), `c1a36daa` (harness CUDA fix), `7930df51` (MoE microbench), `2e84b994`/`253cf76f`/`f9ffb6fb` (Bailing rescue + strip, OPT-08). Use `git show <sha> -- <file>` to see the exact original diff.
## OPT-01 — Merged up/gate expert projections (loader)
@ -90,6 +90,18 @@
- **`tests/test-moe-perf.cpp`** (140 lines): MoE GEMM microbench for A/B without loading a 90GB+ model. Build-only target; run before/after OPT-03/04/05.
- **Rule:** any new arch or kernel gets harness coverage before server runs. Harness green (42/42 CPU + CUDA) is the merge gate.
## OPT-08 — Bailing/Ling trapped tool-call rescue + strip (server correctness, not perf)
- **Goal:** thinking models of the Bailing family (Ling-3.0-Flash) intermittently emit `<tool_call>` arg_key/arg_value XML inside the thinking block. The PEG layer consumes that region as reasoning_content before the tool stage, so the call is lost (empty content, no structured calls) and agent loops stall into empty-response nudges. Recover well-formed blocks post-parse; strip executed blocks from reasoning (vLLM Ling3 terminator parity: reasoning ends where the call begins).
- **Exempt from the PP/TG gate.** Correctness item: validate via HTTP replay below, not benches. Revert is one commit per sub-item (`2e84b994` rescue, `253cf76f` strip-on-rescue, `f9ffb6fb` strip-whenever-calls-exist).
- **Locate:** `examples/server/server-context.cpp`, `server_slot::update_chat_msg`. Anchor: `new_msg.set_tool_call_ids(generated_tool_call_ids, gen_tool_call_id);` inside `if (!new_msg.empty())`.
- **Transform:**
1. New file `examples/server/parsers/bailing_parser.hpp` (namespace `bailing`): `parse_tool_calls(text)` (complete `<tool_call>...</tool_call>` blocks only, bare-identifier names, raw-string args, 8-block cap, never throws), `has_complete_block(text)`, `remove_executed_blocks(text)` (complete blocks only, truncated fragments survive). Include as `<nlohmann/json.hpp>`, matching `server-common.h`, **not** quoted `json.hpp` (fails the server TU build).
2. In `update_chat_msg`, before the anchor line, insert the rescue: on final (non-partial) parses only, when structured calls are absent and content is empty, promote parsed blocks to `new_msg.tool_calls` (ids flow through the existing `set_tool_call_ids` call). Log `Bailing rescue: promoted %d tool call(s)`.
3. Same place, after the rescue: on any final turn carrying structured calls, strip complete blocks from `new_msg.reasoning_content` (covers PEG-parsed turns too, which keep block text in reasoning). Log `Bailing strip: removed ...`. Reasoning with no calls is never touched. See `git show 2e84b994`, `253cf76f`, `f9ffb6fb` for the exact hunks.
- **New arch checklist:** any future thinking template whose tool-call markup can land inside its thinking region gets the same treatment: a `parsers/<family>_parser.hpp` plus the two hooks. If upstream ever ships a dedicated Bailing PEG builder in `common_chat_try_specialized_template`, re-evaluate whether the post-parse fallback is still needed (keep whichever fires; they are compatible, rescue is dormant when structured calls arrive clean).
- **Verify:** standalone harness `examples/server/test-bailing-parser.cpp` (g++ syntax + run, no full build): well-formed trapped call converts with correct name/args, truncated/garbage/plain-text yield nothing, strip preserves narration and truncated fragments. Live: faithful HTTP replay (44k system prompt + real tool history, 8 rounds) must return structured `tool_calls` every round with no `<tool_call>` text in final reasoning_content. Server log must show `Bailing rescue`/`Bailing strip` lines on trapped turns only.
- **Known limitation (do not re-litigate without new evidence):** streaming partial deltas carry raw XML as generated; the final-message strip cannot retract already-streamed text. Streaming clients that accumulate reasoning from deltas will still display it. Fixed only by client-side display scrub or by the model emitting elsewhere (weights).
## Validation protocol (every item)
1. **Build (glove-fit, canonical):**