# hybrid-llama

CPU-inference fork of llama.cpp that combines two independent optimization layers:

- **TurboQuant** (base, in `turboquant/`): AtomicBot-ai fork with KV-cache
  quantization (turbo2/turbo3/turbo4 types), bailingmoe3 bugfixes (tool calls,
  SwiGLU clamps), and MTP/NextN speculative decoding.
- **IQK** (merged in): ik_llama's CPU GEMM optimization layer
  (`turboquant/ggml/src/iqk/`) with SIMD-tuned matmul kernels, fast
  activation quantization, and CPU flash attention.

Goal: ~5x prefill speedup on Ling 3.0 Flash (bailingmoe3), e.g. ~60 t/s to
~300 t/s on a 16-thread AVX2 CPU, with decode unchanged (~9-11 t/s on DDR4)
and all TurboQuant features intact.

The two layers do not conflict: TurboQuant changes sit above the
`ggml_mul_mat` primitive, while IQK accelerates that primitive itself.

## Layout

- `plan.md` - full merge handoff doc (design, porting notes, risks, checklist)
- `turboquant/` - buildable project (TurboQuant + IQK merge)
- `turboquant/ggml/src/iqk/` - IQK sources (from ik_llama)

Upstream base commits (for traceability):

- TurboQuant: `AtomicBot-ai/atomic-llama-cpp-turboquant @ cd5609390`
- IQK: `ikawrakow/ik_llama.cpp @ fe215a8c` (`ggml/src/iqk` only)

## Requirements

- Linux x86_64 with AVX2 (Zen 3 or newer recommended; AVX512 VNNI used when
  present), or ARM with dotprod/MATMUL_INT8
- CMake 3.14+, GCC 11+ or Clang 14+, `make` or Ninja
- Python 3 (only for GGUF conversion scripts)
- Optional: CUDA toolkit (for `-DGGML_CUDA=ON` hybrid GPU+CPU builds)

## Build

```bash
cd turboquant

# CPU-only with IQK (recommended for prefill benchmark)
cmake -B build -DGGML_IQK_MUL_MAT=ON -DGGML_IQK_FLASH_ATTENTION=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

# GPU + CPU hybrid
cmake -B build -DGGML_CUDA=ON -DGGML_IQK_MUL_MAT=ON \
  -DGGML_IQK_FLASH_ATTENTION=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

# Stock TurboQuant behavior (IQK disabled, default)
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)
```

CMake options:

| Option | Default | Effect |
|---|---|---|
| `GGML_IQK_MUL_MAT` | OFF | IQK GEMM kernels + `iqk_quantize_any` fast path in `ggml_compute_forward_mul_mat` |
| `GGML_IQK_FLASH_ATTENTION` | OFF | IQK CPU flash-attention kernels |

Node fusion from ik_llama is intentionally skipped (keeps the stock
`ggml_compute_forward` signature); nearly all of the prefill gain comes from
the GEMM kernels themselves.

## Verify

```bash
cd turboquant

# 1. IQK active (look for IQK init messages)
./build/bin/llama-cli -m <model.gguf> -p "Hello" -n 1 -lv 3 2>&1 | grep -i iqk

# 2. CPU-only prefill benchmark (expect pp512 ~60 -> ~300 t/s)
./build/bin/llama-bench -m <model.gguf> -ngl 0 -t 16

# 3. GPU + CPU hybrid
./build/bin/llama-bench -m <model.gguf> -ngl auto -t 16

# 4. Full server test with Ling 3.0 Flash
./build/bin/llama-server \
  -m Ling-3.0-flash-Q4_K_S.gguf \
  --jinja -ngl 99 -c 32768 \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  --host 127.0.0.1 --port 8080
```

Checklist: model loads with no `blk.0.ssm_f.weight` errors, decode stays at
~9-11 t/s, tool calls and turbo3/turbo4 KV-cache types still work,
`/no_think` and multi-slot (`-np > 1`) behave as before.

## Notes

- IQK repacked weight types (`Q8_2_X4`, `Q8_K_R8/R16`, `*_R4/R8`) are
  intermediate-only: standard GGUFs (Q4_K_S, etc.) load unchanged, and
  activations are quantized to the repacked format at runtime.
- `GGML_TYPE_Q1_0_G128` is aliased to this fork's `GGML_TYPE_Q1_0` (same
  128-bit layout).
- See `plan.md` for the complete porting record, known risks (SIMD
  selection, `GATED_DELTA_NET` guard, threadpool barrier), and the file
  inventory of this merge.