nicolas
/
hybrid-llama
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
hybrid-llama/turboquant/examples/parallel
History
Marvin 1dd0700988 hybrid-llama: merge ik_llama IQK CPU GEMM into TurboQuant fork 
Base: AtomicBot-ai/atomic-llama-cpp-turboquant @ cd5609390. IQK source: ikawrakow/ik_llama.cpp @ fe215a8c (ggml/src/iqk only).

- GGML_IQK_MUL_MAT / GGML_IQK_FLASH_ATTENTION options (default OFF)

- 57 IQK repacked types, blocks, traits; IQK hooks in ggml_compute_forward_mul_mat

- ggml-cpu with IQK ON builds and links; IQK OFF build unaffected

Assisted-by: opencode (Muse Spark)
 		2026-09-05 18:09:49 -03:00
..
CMakeLists.txt hybrid-llama: merge ik_llama IQK CPU GEMM into TurboQuant fork 2026-09-05 18:09:49 -03:00
README.md hybrid-llama: merge ik_llama IQK CPU GEMM into TurboQuant fork 2026-09-05 18:09:49 -03:00
parallel.cpp hybrid-llama: merge ik_llama IQK CPU GEMM into TurboQuant fork 2026-09-05 18:09:49 -03:00

README.md

llama.cpp/example/parallel

Simplified simulation of serving incoming requests in parallel

Example

Generate 128 client requests (-ns 128), simulating 8 concurrent clients (-np 8). The system prompt is shared (-pps), meaning that it is computed once at the start. The client requests consist of up to 10 junk questions (--junk 10) followed by the actual question.

llama-parallel -m model.gguf -np 8 -ns 128 --top-k 1 -pps --junk 10 -c 16384

[!NOTE] It's recommended to use base models with this example. Instruction tuned models might not be able to properly follow the custom chat template specified here, so the results might not be as expected.