docs: OPT-08 Bailing rescue+strip porting entry with rebase anchors
This commit is contained in:
parent
f9ffb6fb2d
commit
617732791e
|
|
@ -1,6 +1,6 @@
|
|||
# Agent porting guide: re-applying the Ampere+Zen3 optimizations
|
||||
|
||||
**Use case:** upstream `ik_llama.cpp` has moved (new model arch, refactored file, new kernels) and you must re-apply this branch's optimizations to the new tree. Work top-down: OPT-01 → OPT-07. Each item is self-contained: *locate* (exact anchor), *transform* (exact edit), *verify*.
|
||||
**Use case:** upstream `ik_llama.cpp` has moved (new model arch, refactored file, new kernels) and you must re-apply this branch's optimizations to the new tree. Work top-down: OPT-01 → OPT-08. Each item is self-contained: *locate* (exact anchor), *transform* (exact edit), *verify*.
|
||||
|
||||
**Target rig (do not re-tune for other hardware):** RTX 3090 24GB + RTX 3070 8GB (`sm_86`, no P2P) + Ryzen 5900XT 16C/32T (`znver3`, DDR4-3600). PP-bound long-context agentic sessions. Glove-fit non-portable builds are acceptable and expected.
|
||||
|
||||
|
|
@ -10,7 +10,7 @@
|
|||
2. **New CUDA kernels are opt-in env flags, default off**, until A/B proves a win. Pattern: `getenv("DELTA_WY_CUDA")` → branch → exact fallback. See OPT-06.
|
||||
3. **One optimization = one commit**, message prefix: `qwen4exp:`, `cuda:`, `tests:`, `ggml:`.
|
||||
4. **Validate after every item** (see "Validation protocol"). If the anchor is gone (upstream refactor), locate the successor by searching the anchor's keywords, adapt the transform, and note the new anchor in the commit message.
|
||||
5. **Reference implementation:** commits `e2728c85` (CPU/graph), `a47a8a65`/`f5b494b0`/`3a7d1016` (harness + WY candidate), `a34feeb0` (CUDA kernel), `c1a36daa` (harness CUDA fix), `7930df51` (MoE microbench). Use `git show <sha> -- <file>` to see the exact original diff.
|
||||
5. **Reference implementation:** commits `e2728c85` (CPU/graph), `a47a8a65`/`f5b494b0`/`3a7d1016` (harness + WY candidate), `a34feeb0` (CUDA kernel), `c1a36daa` (harness CUDA fix), `7930df51` (MoE microbench), `2e84b994`/`253cf76f`/`f9ffb6fb` (Bailing rescue + strip, OPT-08). Use `git show <sha> -- <file>` to see the exact original diff.
|
||||
|
||||
## OPT-01 — Merged up/gate expert projections (loader)
|
||||
|
||||
|
|
@ -90,6 +90,18 @@
|
|||
- **`tests/test-moe-perf.cpp`** (140 lines): MoE GEMM microbench for A/B without loading a 90GB+ model. Build-only target; run before/after OPT-03/04/05.
|
||||
- **Rule:** any new arch or kernel gets harness coverage before server runs. Harness green (42/42 CPU + CUDA) is the merge gate.
|
||||
|
||||
## OPT-08 — Bailing/Ling trapped tool-call rescue + strip (server correctness, not perf)
|
||||
- **Goal:** thinking models of the Bailing family (Ling-3.0-Flash) intermittently emit `<tool_call>` arg_key/arg_value XML inside the thinking block. The PEG layer consumes that region as reasoning_content before the tool stage, so the call is lost (empty content, no structured calls) and agent loops stall into empty-response nudges. Recover well-formed blocks post-parse; strip executed blocks from reasoning (vLLM Ling3 terminator parity: reasoning ends where the call begins).
|
||||
- **Exempt from the PP/TG gate.** Correctness item: validate via HTTP replay below, not benches. Revert is one commit per sub-item (`2e84b994` rescue, `253cf76f` strip-on-rescue, `f9ffb6fb` strip-whenever-calls-exist).
|
||||
- **Locate:** `examples/server/server-context.cpp`, `server_slot::update_chat_msg`. Anchor: `new_msg.set_tool_call_ids(generated_tool_call_ids, gen_tool_call_id);` inside `if (!new_msg.empty())`.
|
||||
- **Transform:**
|
||||
1. New file `examples/server/parsers/bailing_parser.hpp` (namespace `bailing`): `parse_tool_calls(text)` (complete `<tool_call>...</tool_call>` blocks only, bare-identifier names, raw-string args, 8-block cap, never throws), `has_complete_block(text)`, `remove_executed_blocks(text)` (complete blocks only, truncated fragments survive). Include as `<nlohmann/json.hpp>`, matching `server-common.h`, **not** quoted `json.hpp` (fails the server TU build).
|
||||
2. In `update_chat_msg`, before the anchor line, insert the rescue: on final (non-partial) parses only, when structured calls are absent and content is empty, promote parsed blocks to `new_msg.tool_calls` (ids flow through the existing `set_tool_call_ids` call). Log `Bailing rescue: promoted %d tool call(s)`.
|
||||
3. Same place, after the rescue: on any final turn carrying structured calls, strip complete blocks from `new_msg.reasoning_content` (covers PEG-parsed turns too, which keep block text in reasoning). Log `Bailing strip: removed ...`. Reasoning with no calls is never touched. See `git show 2e84b994`, `253cf76f`, `f9ffb6fb` for the exact hunks.
|
||||
- **New arch checklist:** any future thinking template whose tool-call markup can land inside its thinking region gets the same treatment: a `parsers/<family>_parser.hpp` plus the two hooks. If upstream ever ships a dedicated Bailing PEG builder in `common_chat_try_specialized_template`, re-evaluate whether the post-parse fallback is still needed (keep whichever fires; they are compatible, rescue is dormant when structured calls arrive clean).
|
||||
- **Verify:** standalone harness `examples/server/test-bailing-parser.cpp` (g++ syntax + run, no full build): well-formed trapped call converts with correct name/args, truncated/garbage/plain-text yield nothing, strip preserves narration and truncated fragments. Live: faithful HTTP replay (44k system prompt + real tool history, 8 rounds) must return structured `tool_calls` every round with no `<tool_call>` text in final reasoning_content. Server log must show `Bailing rescue`/`Bailing strip` lines on trapped turns only.
|
||||
- **Known limitation (do not re-litigate without new evidence):** streaming partial deltas carry raw XML as generated; the final-message strip cannot retract already-streamed text. Streaming clients that accumulate reasoning from deltas will still display it. Fixed only by client-side display scrub or by the model emitting elsewhere (weights).
|
||||
|
||||
## Validation protocol (every item)
|
||||
|
||||
1. **Build (glove-fit, canonical):**
|
||||
|
|
|
|||
Loading…
Reference in New Issue