16 KiB

Raw Blame History

IQK + TurboQuant Merge: Complete Coding Handoff

Objective

Merge ik_llama's IQK CPU GEMM optimization layer into the AtomicBot-ai/atomic-llama-cpp-turboquant fork to get ~5x prefill speedup on Ling 3.0 Flash (bailingmoe3) while retaining TurboQuant's bugfixes (tool calls, token handling, KV cache compression).

Repository URLs

TurboQuant (target): https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant
ik_llama (source): https://github.com/ikawrakow/ik_llama.cpp

Expected Outcome

Prefill: 60 t/s → ~300 t/s (5x improvement)
Decode: unchanged (~9-11 t/s on DDR4)
All TurboQuant features intact (KV cache compression, bailingmoe3 fixes, MTP/NextN)

Architecture Comparison

Aspect	TurboQuant	ik_llama
Code structure	Modular (separate backend libs)	Monolithic (single ggml.c)
CPU backend file	`ggml/src/ggml-cpu/ggml-cpu.c` (4K lines)	`ggml/src/ggml.c` (32K lines)
Forward dispatch	`void ggml_compute_forward(params, tensor)`	`int ggml_compute_forward(params, tensor, cgraph, node_n)`
Params struct	`params->threadpool` (ggml-cpu-impl.h)	`params->shared` (inline in ggml.c)
Type traits	`type_traits_cpu[]`	`type_traits[]`
IQK present?	No	Yes
KV cache compression	Yes (TURBO types)	No
bailingmoe3	Yes (bugfixed)	May have issues

Why They Don't Conflict

The two forks modify different layers:

TurboQuant: Higher-level (KV cache quantization types, architecture-specific fixes, MTP speculative decoding)
IQK: Lower-level (CPU GEMM primitives in ggml/src/iqk/)

IQK accelerates ggml_mul_mat — the primitive every model calls. TurboQuant's changes are above that layer. The KV cache compression (TURBO types) and Hadamard fast path are in different code regions.

Step 1: Copy IQK Source Files

Copy the entire ggml/src/iqk/ directory from ik_llama into TurboQuant:

# From the TurboQuant repo root
git remote add ik_llama https://github.com/ikawrakow/ik_llama.cpp
git fetch ik_llama
git checkout ik_llama/master -- ggml/src/iqk/

This brings in 36 files (~2.1 MB total):

Core files (always compiled):

iqk/iqk_config.h (1.7 KB) — build config
iqk/iqk_common.h (42 KB) — shared utilities
iqk/iqk_utils.h (12 KB) — helper macros
iqk/iqk_quantize.cpp (439 KB) — weight quantization
iqk/iqk_quantize.h (34 KB) — quantization API
iqk/iqk_cpu_ops.cpp (43 KB) — SIMD-accelerated CPU ops (top-k, argsort, etc.)
iqk/iqk_cpu_ops.h (2 KB) — ops API

GEMM files (compiled when GGML_IQK_MUL_MAT=ON):

iqk/iqk_mul_mat.cpp (99 KB) — main dispatch + node fusion
iqk/iqk_mul_mat.h (4.5 KB) — matmul API
iqk/iqk_kda.cpp (17 KB) — KDA (Gated Delta Net) fused path
iqk/iqk_flash_attn.cpp (32 KB) — CPU flash attention
iqk/iqk_flash_impl.h (2 KB) — FA implementation
iqk/iqk_gemm_kquants.cpp (250 KB) — Q2_K through Q6_K kernels
iqk/iqk_gemm_kquants.h (0.4 KB)
iqk/iqk_gemm_ktquants.cpp (126 KB) — Trellis quant kernels
iqk/iqk_gemm_ktquants.h (0.3 KB)
iqk/iqk_gemm_iquants.cpp (202 KB) — IQ2_XXS through IQ4_XS kernels
iqk/iqk_gemm_iquants.h (0.3 KB)
iqk/iqk_gemm_iqk_quants.cpp (288 KB) — IQ2_K through IQ6_K kernels
iqk/iqk_gemm_iqk_quants.h (0.3 KB)
iqk/iqk_gemm_legacy_quants.cpp (172 KB) — Q4_0, Q4_1, Q5_0, Q5_1, Q8_0
iqk/iqk_gemm_legacy_quants.h (0.5 KB)
iqk/iqk_gemm_floats.cpp (48 KB) — FP16/FP32/BF16 kernels
iqk/iqk_gemm_floats.h (0.3 KB)
iqk/iqk_gemm_1bit.cpp (188 KB) — Bitnet kernels
iqk/iqk_gemm_1bit.h (0.3 KB)

Flash attention template files:

iqk/fa/iqk_fa_templates.h — FA template definitions
iqk/fa/iqk_fa_64_64.cpp through iqk_fa_576_512.cpp (9 files, ~1.6 KB each) — size-specialized FA kernels

Step 2: CMake Integration

2a. Add cmake option

In ggml/src/CMakeLists.txt (or create a new section), add:

option(GGML_IQK_MUL_MAT "Enable optimized IQK matrix multiplications" OFF)
option(GGML_IQK_FLASH_ATTENTION "Enable IQK Flash Attention kernels" OFF)

2b. Add IQK sources to ggml-cpu backend

In ggml/src/ggml-cpu/CMakeLists.txt, inside the ggml_add_cpu_backend_variant_impl() function, before the target_sources(${GGML_CPU_NAME} PRIVATE ${GGML_CPU_SOURCES}) line, add:

# IQK sources — unconditional (quantize + CPU ops)
set(GGML_CPU_IQK_SOURCES
    ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_quantize.cpp
    ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_cpu_ops.cpp
)
set(GGML_CPU_IQK_HEADERS
    ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_config.h
    ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_common.h
    ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_utils.h
    ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_quantize.h
    ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_cpu_ops.h
)

if(GGML_IQK_MUL_MAT)
    list(APPEND GGML_CPU_IQK_SOURCES
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_mul_mat.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_kda.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_flash_attn.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_gemm_floats.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_gemm_kquants.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_gemm_ktquants.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_gemm_iquants.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_gemm_iqk_quants.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_gemm_1bit.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_gemm_legacy_quants.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/fa/iqk_fa_64_64.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/fa/iqk_fa_96_96.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/fa/iqk_fa_128_128.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/fa/iqk_fa_192_128.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/fa/iqk_fa_192_192.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/fa/iqk_fa_256_256.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/fa/iqk_fa_320_256.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/fa/iqk_fa_512_512.cpp
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/fa/iqk_fa_576_512.cpp
    )
    list(APPEND GGML_CPU_IQK_HEADERS
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_mul_mat.h
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_flash_impl.h
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/fa/iqk_fa_templates.h
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_gemm_floats.h
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_gemm_kquants.h
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_gemm_ktquants.h
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_gemm_iquants.h
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_gemm_iqk_quants.h
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_gemm_1bit.h
        ${CMAKE_CURRENT_SOURCE_DIR}/../iqk/iqk_gemm_legacy_quants.h
    )
endif()

target_sources(${GGML_CPU_NAME} PRIVATE
    ${GGML_CPU_SOURCES}
    ${GGML_CPU_IQK_SOURCES}
)
target_sources(${GGML_CPU_NAME} PRIVATE FILE_SET HEADERS
    ${GGML_CPU_HEADERS}
    ${GGML_CPU_IQK_HEADERS}
)
target_include_directories(${GGML_CPU_NAME} PRIVATE
    ${CMAKE_CURRENT_SOURCE_DIR}/../iqk
)

2c. Add compile definitions

In the same ggml_add_cpu_backend_variant_impl() function:

if(GGML_IQK_MUL_MAT)
    target_compile_definitions(${GGML_CPU_NAME} PRIVATE GGML_USE_IQK_MULMAT)
endif()
if(GGML_IQK_FLASH_ATTENTION)
    target_compile_definitions(${GGML_CPU_NAME} PRIVATE GGML_IQK_FLASH_ATTENTION)
endif()

2d. Top-level CMakeLists.txt

In the root CMakeLists.txt, add the options and pass them through:

option(GGML_IQK_MUL_MAT "Enable optimized IQK matrix multiplications" OFF)
option(GGML_IQK_FLASH_ATTENTION "Enable IQK Flash Attention kernels" OFF)

And ensure they're passed to the ggml subdirectory via add_subdirectory(ggml) or set().

Step 3: Adapt the Forward Dispatch (THE HARD PART)

This is where the structural mismatch matters. In ik_llama, IQK hooks into ggml_compute_forward_mul_mat() in ggml.c. In TurboQuant, that function lives in ggml-cpu/ggml-cpu.c with a different signature.

3a. Locate the dispatch function

In TurboQuant's ggml/src/ggml-cpu/ggml-cpu.c, find ggml_compute_forward_mul_mat(). This is where MUL_MAT operations are dispatched.

3b. Port the IQK hooks

From ik_llama's ggml.c lines 18016-18111, port these three code blocks into TurboQuant's ggml_compute_forward_mul_mat():

Block 1: Early exit fast path (ik_llama lines 18016-18024)

#ifdef GGML_USE_IQK_MULMAT
    // When src1 is already in vec_dot_type and dst is F32, use IQK directly
    if (src1->type == vec_dot_type && dst->type == GGML_TYPE_F32) {
        iqk_mul_mat_4d(ne01, ne11, ne00, ne02, ne03, ne12, ne13,
                       src0->data, src1->data, dst->data,
                       src0->type, vec_dot_type,
                       params->ith, params->nth);
        return;
    }
#endif

Block 2: Custom quantization (ik_llama lines 18039-18047)

#ifdef GGML_USE_IQK_MULMAT
    // Replace standard from_float with IQK's faster quantization
    if (src1->type != vec_dot_type) {
        iqk_quantize_any(src1->type, vec_dot_type, ne10, ne11, ne12, ne13,
                         src1->data, wdata, params->ith, params->nth);
    }
#else
    // Original llama.cpp quantization path
    ggml_quantize_mat(src1->type, vec_dot_type, ne10, ne11, ne12, ne13,
                      src1->data, wdata, params->ith, params->nth);
#endif

Block 3: Main matmul + node fusion (ik_llama lines 18083-18111)

#ifdef GGML_USE_IQK_MULMAT
    iqk_mul_mat_4d(ne01, ne11, ne00, ne02, ne03, ne12, ne13,
                   src0->data, wdata, dst->data,
                   vec_dot_type, GGML_TYPE_F32,
                   params->ith, params->nth);
#else
    // Original ggml_compute_forward_mul_mat_batched path
    ...
#endif

3c. Adapt params->shared to params->threadpool

ik_llama's IQK uses params->shared for barrier and abort callbacks. TurboQuant uses params->threadpool. You need to:

Check if iqk_mul_mat_4d() and related functions actually use params->shared (they likely don't for the basic path — the shared struct is mainly for synchronization in multi-threaded dispatch)
If they do, create an adapter that wraps params->threadpool into the interface IQK expects
If they don't, this is a non-issue

3d. Adapt type_traits

ik_llama modifies type_traits[] entries to use GGML_TYPE_Q8_2_X4 as vec_dot_type for q4_0, q4_1, q5_0, q5_1, q6_0, q8_0 (ik_llama ggml.c lines 714-856).

In TurboQuant, the equivalent is type_traits_cpu[] in ggml-cpu/ggml-cpu.c or ggml-cpu/traits.cpp. Add the same modifications:

#ifdef GGML_USE_IQK_MULMAT
    // Override vec_dot_type for legacy quants to use IQK's repacked format
    type_traits_cpu[GGML_TYPE_Q4_0].vec_dot_type = GGML_TYPE_Q8_2_X4;
    type_traits_cpu[GGML_TYPE_Q4_1].vec_dot_type = GGML_TYPE_Q8_2_X4;
    type_traits_cpu[GGML_TYPE_Q5_0].vec_dot_type = GGML_TYPE_Q8_2_X4;
    type_traits_cpu[GGML_TYPE_Q5_1].vec_dot_type = GGML_TYPE_Q8_2_X4;
    type_traits_cpu[GGML_TYPE_Q6_0].vec_dot_type = GGML_TYPE_Q8_2_X4;
    type_traits_cpu[GGML_TYPE_Q8_0].vec_dot_type = GGML_TYPE_Q8_2_X4;
#endif

Step 4: Add IQK Includes

In ggml-cpu/ggml-cpu.c, add at the top:

#ifdef GGML_USE_IQK_MULMAT
#include "iqk/iqk_quantize.h"
#include "iqk/iqk_cpu_ops.h"
#include "iqk/iqk_mul_mat.h"
#include "iqk/iqk_config.h"
#endif

Step 5: Node Fusion (Optional, Advanced)

ik_llama fuses consecutive MUL_MAT nodes sharing the same src1 (lines 18083-18111). This requires the int ggml_compute_forward() signature that returns an updated node index.

Option A (recommended): Skip node fusion initially. Get the basic IQK fast path working first. The 5x prefill speedup comes primarily from the optimized GEMM kernels, not from node fusion. Node fusion adds maybe 10-20% on top.

Option B (full port): Modify TurboQuant's forward dispatch signature to match ik_llama's. This requires:

Changing void ggml_compute_forward(...) to int ggml_compute_forward(...) in ggml-cpu.c
Updating the dispatch loop in ggml-cpu.c to track node_n
Updating all callers of ggml_compute_forward

This is invasive and risky. Do it only after the basic path works.

Option C (hybrid): Use TurboQuant's existing ggml_cpu_try_fuse_ops() mechanism (line 3162 of ggml-cpu.c) to implement IQK's MUL_MAT fusion as a separate fusion pass. This avoids changing the forward dispatch signature.

Step 6: Build and Test

cd atomic-llama-cpp-turboquant
cmake -B build -DGGML_CUDA=ON -DGGML_IQK_MUL_MAT=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

Test 1: Verify IQK is active

./build/bin/llama-cli -m <model.gguf> -p "Hello" -n 1 -lv 3 2>&1 | grep -i iqk

Should show IQK initialization messages.

Test 2: CPU-only prefill benchmark

./build/bin/llama-bench -m <model.gguf> -ngl 0 -t 16

Compare pp512 before and after. Expected: 60 t/s → ~300 t/s.

Test 3: GPU + CPU hybrid

./build/bin/llama-bench -m <model.gguf> -ngl auto -t 16

Test 4: Full server test with Ling 3.0 Flash

./build/bin/llama-server \
  -m Ling-3.0-flash-Q4_K_S.gguf \
  --jinja -ngl 99 -c 32768 \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  --host 127.0.0.1 --port 8080

Verify:

Model loads without errors (especially blk.0.ssm_f.weight)
Tool calls work correctly (TurboQuant's bugfix)
KV cache compression works (TurboQuant's turbo3/turbo4 types)
Prefill speed is ~300 t/s
Decode speed is ~9-11 t/s (DDR4 bandwidth limit)

Known Risks

GATED_DELTA_NET: Both forks have this op but implementations may differ. ik_llama has iqk_fused_delta_net() which is called when IQK is available. Verify this path is correctly guarded by #ifdef GGML_USE_IQK_MULMAT and doesn't conflict with TurboQuant's implementation.
SIMD detection: IQK uses compile-time SIMD detection (__AVX2__, __AVX512__, __aarch64__). TurboQuant's build system may set these differently. Verify the right code paths are selected.
Repacked quantization types: IQK introduces Q8_K_R8, Q8_K_R16 as first-class GGUF types. These need to be registered in TurboQuant's type system. Check if ggml-common.h in TurboQuant already has these or needs them added.
Thread count: IQK's tiling strategy assumes certain thread counts. The func16 path (16-row interleaved) is only used on AVX512. On AVX2 (your Zen 3), the funcs[] array is used instead. Verify the dispatch selects the right path.

File Inventory

Files to CREATE (new):

ggml/src/iqk/ — entire directory (36 files, ~2.1 MB)
ggml/src/ggml-cpu/CMakeLists.txt — modified (add IQK sources)

Files to MODIFY:

ggml/src/CMakeLists.txt — add GGML_IQK_MUL_MAT option
CMakeLists.txt (root) — add option passthrough
ggml/src/ggml-cpu/ggml-cpu.c — add IQK hooks in ggml_compute_forward_mul_mat()

Files to NOT TOUCH:

ggml/src/ggml-turbo-quant.c — KV cache compression (orthogonal)
ggml/src/ggml-cpu/ggml-cpu-impl.h — params struct (no change needed)
Any MTP/NextN files — speculative decoding (orthogonal)
Any bailingmoe3 architecture files — model-specific (orthogonal)

Time Estimate

Step 1-2 (copy + cmake): 30 minutes
Step 3 (forward dispatch adaptation): 2-4 hours (the hard part)
Step 4-5 (includes + optional fusion): 30 minutes
Step 6 (build + test): 1-2 hours
Total: 4-7 hours for a competent C/C++ developer

Verification Checklist

cmake -DGGML_IQK_MUL_MAT=ON configures without errors
Build completes without errors
IQK initialization message appears in verbose logs
Ling 3.0 Flash loads without blk.0.ssm_f.weight errors
Prefill speed improves from ~60 to ~300 t/s
Decode speed is unchanged (~9-11 t/s)
Tool calls work correctly
KV cache compression (turbo3/turbo4) works
/no_think parameter works
Multi-slot (-np > 1) works if previously working

16 KiB Raw Blame History