3.6 KiB

Raw Blame History

hybrid-llama

CPU-inference fork of llama.cpp that combines two independent optimization layers:

TurboQuant (base, in turboquant/): AtomicBot-ai fork with KV-cache quantization (turbo2/turbo3/turbo4 types), bailingmoe3 bugfixes (tool calls, SwiGLU clamps), and MTP/NextN speculative decoding.
IQK (merged in): ik_llama's CPU GEMM optimization layer (turboquant/ggml/src/iqk/) with SIMD-tuned matmul kernels, fast activation quantization, and CPU flash attention.

Goal: ~5x prefill speedup on Ling 3.0 Flash (bailingmoe3), e.g. ~60 t/s to ~300 t/s on a 16-thread AVX2 CPU, with decode unchanged (~9-11 t/s on DDR4) and all TurboQuant features intact.

The two layers do not conflict: TurboQuant changes sit above the ggml_mul_mat primitive, while IQK accelerates that primitive itself.

Layout

plan.md - full merge handoff doc (design, porting notes, risks, checklist)
turboquant/ - buildable project (TurboQuant + IQK merge)
turboquant/ggml/src/iqk/ - IQK sources (from ik_llama)

Upstream base commits (for traceability):

TurboQuant: AtomicBot-ai/atomic-llama-cpp-turboquant @ cd5609390
IQK: ikawrakow/ik_llama.cpp @ fe215a8c (ggml/src/iqk only)

Requirements

Linux x86_64 with AVX2 (Zen 3 or newer recommended; AVX512 VNNI used when present), or ARM with dotprod/MATMUL_INT8
CMake 3.14+, GCC 11+ or Clang 14+, make or Ninja
Python 3 (only for GGUF conversion scripts)
Optional: CUDA toolkit (for -DGGML_CUDA=ON hybrid GPU+CPU builds)

Build

cd turboquant

# CPU-only with IQK (recommended for prefill benchmark)
cmake -B build -DGGML_IQK_MUL_MAT=ON -DGGML_IQK_FLASH_ATTENTION=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

# GPU + CPU hybrid
cmake -B build -DGGML_CUDA=ON -DGGML_IQK_MUL_MAT=ON \
  -DGGML_IQK_FLASH_ATTENTION=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

# Stock TurboQuant behavior (IQK disabled, default)
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

CMake options:

Option	Default	Effect
`GGML_IQK_MUL_MAT`	OFF	IQK GEMM kernels + `iqk_quantize_any` fast path in `ggml_compute_forward_mul_mat`
`GGML_IQK_FLASH_ATTENTION`	OFF	IQK CPU flash-attention kernels

Node fusion from ik_llama is intentionally skipped (keeps the stock ggml_compute_forward signature); nearly all of the prefill gain comes from the GEMM kernels themselves.

Verify

cd turboquant

# 1. IQK active (look for IQK init messages)
./build/bin/llama-cli -m <model.gguf> -p "Hello" -n 1 -lv 3 2>&1 | grep -i iqk

# 2. CPU-only prefill benchmark (expect pp512 ~60 -> ~300 t/s)
./build/bin/llama-bench -m <model.gguf> -ngl 0 -t 16

# 3. GPU + CPU hybrid
./build/bin/llama-bench -m <model.gguf> -ngl auto -t 16

# 4. Full server test with Ling 3.0 Flash
./build/bin/llama-server \
  -m Ling-3.0-flash-Q4_K_S.gguf \
  --jinja -ngl 99 -c 32768 \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  --host 127.0.0.1 --port 8080

Checklist: model loads with no blk.0.ssm_f.weight errors, decode stays at ~9-11 t/s, tool calls and turbo3/turbo4 KV-cache types still work, /no_think and multi-slot (-np > 1) behave as before.

Notes

IQK repacked weight types (Q8_2_X4, Q8_K_R8/R16, *_R4/R8) are intermediate-only: standard GGUFs (Q4_K_S, etc.) load unchanged, and activations are quantized to the repacked format at runtime.
GGML_TYPE_Q1_0_G128 is aliased to this fork's GGML_TYPE_Q1_0 (same 128-bit layout).
See plan.md for the complete porting record, known risks (SIMD selection, GATED_DELTA_NET guard, threadpool barrier), and the file inventory of this merge.

3.6 KiB Raw Blame History