From eaaa66842034b94e3460676b80678c96d9f7ebcc Mon Sep 17 00:00:00 2001
From: Marvin <nsanchez1980@gmail.com>
Date: Sat, 5 Sep 2026 18:11:59 -0300
Subject: [PATCH] Add README with description and build instructions

Assisted-by: opencode (Muse Spark)
---
 README.md | 104 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 104 insertions(+)
 create mode 100644 README.md

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..0dd5f9e
--- /dev/null
+++ b/README.md
@@ -0,0 +1,104 @@
+# hybrid-llama
+
+CPU-inference fork of llama.cpp that combines two independent optimization layers:
+
+- **TurboQuant** (base, in `turboquant/`): AtomicBot-ai fork with KV-cache
+  quantization (turbo2/turbo3/turbo4 types), bailingmoe3 bugfixes (tool calls,
+  SwiGLU clamps), and MTP/NextN speculative decoding.
+- **IQK** (merged in): ik_llama's CPU GEMM optimization layer
+  (`turboquant/ggml/src/iqk/`) with SIMD-tuned matmul kernels, fast
+  activation quantization, and CPU flash attention.
+
+Goal: ~5x prefill speedup on Ling 3.0 Flash (bailingmoe3), e.g. ~60 t/s to
+~300 t/s on a 16-thread AVX2 CPU, with decode unchanged (~9-11 t/s on DDR4)
+and all TurboQuant features intact.
+
+The two layers do not conflict: TurboQuant changes sit above the
+`ggml_mul_mat` primitive, while IQK accelerates that primitive itself.
+
+## Layout
+
+- `plan.md` - full merge handoff doc (design, porting notes, risks, checklist)
+- `turboquant/` - buildable project (TurboQuant + IQK merge)
+- `turboquant/ggml/src/iqk/` - IQK sources (from ik_llama)
+
+Upstream base commits (for traceability):
+
+- TurboQuant: `AtomicBot-ai/atomic-llama-cpp-turboquant @ cd5609390`
+- IQK: `ikawrakow/ik_llama.cpp @ fe215a8c` (`ggml/src/iqk` only)
+
+## Requirements
+
+- Linux x86_64 with AVX2 (Zen 3 or newer recommended; AVX512 VNNI used when
+  present), or ARM with dotprod/MATMUL_INT8
+- CMake 3.14+, GCC 11+ or Clang 14+, `make` or Ninja
+- Python 3 (only for GGUF conversion scripts)
+- Optional: CUDA toolkit (for `-DGGML_CUDA=ON` hybrid GPU+CPU builds)
+
+## Build
+
+```bash
+cd turboquant
+
+# CPU-only with IQK (recommended for prefill benchmark)
+cmake -B build -DGGML_IQK_MUL_MAT=ON -DGGML_IQK_FLASH_ATTENTION=ON \
+  -DCMAKE_BUILD_TYPE=Release
+cmake --build build --config Release -j$(nproc)
+
+# GPU + CPU hybrid
+cmake -B build -DGGML_CUDA=ON -DGGML_IQK_MUL_MAT=ON \
+  -DGGML_IQK_FLASH_ATTENTION=ON -DCMAKE_BUILD_TYPE=Release
+cmake --build build --config Release -j$(nproc)
+
+# Stock TurboQuant behavior (IQK disabled, default)
+cmake -B build -DCMAKE_BUILD_TYPE=Release
+cmake --build build --config Release -j$(nproc)
+```
+
+CMake options:
+
+| Option | Default | Effect |
+|---|---|---|
+| `GGML_IQK_MUL_MAT` | OFF | IQK GEMM kernels + `iqk_quantize_any` fast path in `ggml_compute_forward_mul_mat` |
+| `GGML_IQK_FLASH_ATTENTION` | OFF | IQK CPU flash-attention kernels |
+
+Node fusion from ik_llama is intentionally skipped (keeps the stock
+`ggml_compute_forward` signature); nearly all of the prefill gain comes from
+the GEMM kernels themselves.
+
+## Verify
+
+```bash
+cd turboquant
+
+# 1. IQK active (look for IQK init messages)
+./build/bin/llama-cli -m <model.gguf> -p "Hello" -n 1 -lv 3 2>&1 | grep -i iqk
+
+# 2. CPU-only prefill benchmark (expect pp512 ~60 -> ~300 t/s)
+./build/bin/llama-bench -m <model.gguf> -ngl 0 -t 16
+
+# 3. GPU + CPU hybrid
+./build/bin/llama-bench -m <model.gguf> -ngl auto -t 16
+
+# 4. Full server test with Ling 3.0 Flash
+./build/bin/llama-server \
+  -m Ling-3.0-flash-Q4_K_S.gguf \
+  --jinja -ngl 99 -c 32768 \
+  --temp 0.6 --top-p 0.95 --top-k 20 \
+  --host 127.0.0.1 --port 8080
+```
+
+Checklist: model loads with no `blk.0.ssm_f.weight` errors, decode stays at
+~9-11 t/s, tool calls and turbo3/turbo4 KV-cache types still work,
+`/no_think` and multi-slot (`-np > 1`) behave as before.
+
+## Notes
+
+- IQK repacked weight types (`Q8_2_X4`, `Q8_K_R8/R16`, `*_R4/R8`) are
+  intermediate-only: standard GGUFs (Q4_K_S, etc.) load unchanged, and
+  activations are quantized to the repacked format at runtime.
+- `GGML_TYPE_Q1_0_G128` is aliased to this fork's `GGML_TYPE_Q1_0` (same
+  128-bit layout).
+- See `plan.md` for the complete porting record, known risks (SIMD
+  selection, `GATED_DELTA_NET` guard, threadpool barrier), and the file
+  inventory of this merge.