7945404458
* initial map to load deepseek 4 arch * wip * wip: match graph build and attn logic for dpv4 * wip: Enhance DeepSeek-V4 architecture with new tensor types and sqrtsoftplus gating function * Update DeepSeek-V4 to support raw key indexing with read/write indices * fix mismatch in attn_raw * Enable FA with CSA/HCA * Fix logit mismatch with FA path * Clean traces and logs for debug * Refactor DSV4 tensor handling for MTP execution and improve raw context management * Refactor DeepSeek4 tensor operations: replace manual weighted sum and post-processing with new helper functions * Share mHC pre-projection and fix packed DSV4 writes * DSV4: add shared top-k selection and improve mask handling * Fix DSV4 c2048 view stride and duplicate loader instantiation * Reuse shared RMS normalization in DSV4 graph * Replace DSV4 indexer rotation with shared Hadamard * Share CSA visibility mask with DSV4 LID * dsv4: document dependency ordering and reset state * Remove DSV4 zero-dependency graph shim * Fix DSV4 packed stream execution * Remove DSV4 l_out backend override * Enable DSV4 quantized K-only cache * Revert "Enable DSV4 quantized K-only cache" This reverts commit 04f9b425321f62ba60e16d1bea2f8de714cfe855. * Fix DSV4 quantized cache accounting * Fail closed on unsupported DSV4 cache lifecycle operations * Various optimizations * llama: fix GGML_METAL=ON build - missing ggml-metal.h include in llama-dflash.cpp (#2134) llama-dflash.cpp calls ggml_backend_is_metal() and ggml_backend_metal_set_n_cb() inside an #ifdef GGML_USE_METAL block but never includes ggml-metal.h, so any Metal-enabled build fails to compile. Add the same guarded include llama.cpp already uses. * New op: ggml_sum_rows_ext (#2132) * Add ggml_sum_rows_ext * openPangu: use ggml_sum_rows_ext also in mhc_post * openPangu: use ggml_sum_rows_ext also in mhc_tail * Minor * Reuse shared inverse RoPE operation for DSV4 * Reuse maintainer CUDA concat implementation * WIP * hc_pre * hc_post * Remove unnecessary mask manipulations * WIP * Take into account swiglu limits * Turn on fused indexer by default * Give names to mat mul results * More named ops * dsv4: do not uselessly copy the KV cache +20% TG at 32k tokens * mask_to_index and make CPU FA work with that * Much better CPU-only, CUDA still not functional * Better CPU TG I'm now at 9.7 t/s for zero context and 6.5 t/s for context of 32k. PP is 120 t/s for short context and 101 t/s at 32k. * Even better CPU TG I'm now at 8.1 t/s for context of 32k tokens. * Turn off DSA on CUDA for now * Fix CUDA DSA * Remove again the unnecessary softmax result buffer * Experiments * Various * More named ops * Forgot to uncomment --------- Co-authored-by: samuel <samueloliveira32df@gmail.com> Co-authored-by: hchengit <95317477+hchengit@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| CMakeLists.txt | ||
| README.md | ||
| sweep-bench-plot.py | ||
| sweep-bench.cpp | ||
README.md
ik_llama.cpp/example/sweep-bench
Benchmark the prompt processing and token generation performance of ik_llama.cpp
by doing a sweep over a whole context size and gathering performance metrics
in each ubatch-sized window. Only a single token sequence is used.
The benchmark steps are:
for each ubatch-sized window in context:
1. generate ubatch/4 tokens (not the whole window to save some time)
2. measure generation performance
3. remove generated tokens from KV cache
4. prepare a ubatch-sized batch of random tokens
4. process prepated batch
5. measure prompt processing performance
The purpose of the benchmark is to visualize how the performance changes with the context size without averaging the metrics values over the whole context.
Usage
./llama-sweep-bench -c 8704 -ub 512 -m models/Meta-Llama-3.2-3B-Instruct-Q8_0.gguf
Sample results
PP- prompt tokens per ubatchTG- generated tokens per ubatchN_KV- current KV cache sizeT_PP- prompt processing time (i.e. time to first token)S_PP- prompt processing speed ((B*PP)/T_PPorPP/T_PP)T_TG- time to generate all batchesS_TG- text generation speed ((B*TG)/T_TG)
| PP | TG | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s |
|---|---|---|---|---|---|---|
| 512 | 128 | 0 | 1.100 | 465.51 | 2.311 | 55.38 |
| 512 | 128 | 512 | 1.183 | 432.97 | 1.895 | 67.55 |
| 512 | 128 | 1024 | 1.305 | 392.38 | 2.071 | 61.81 |
| 512 | 128 | 1536 | 1.279 | 400.42 | 2.164 | 59.14 |
| 512 | 128 | 2048 | 1.571 | 325.96 | 2.280 | 56.14 |
| 512 | 128 | 2560 | 1.431 | 357.87 | 2.418 | 52.94 |
| 512 | 128 | 3072 | 1.515 | 337.93 | 2.566 | 49.88 |
| 512 | 128 | 3584 | 1.588 | 322.34 | 2.722 | 47.03 |
| 512 | 128 | 4096 | 1.675 | 305.70 | 2.864 | 44.69 |
| 512 | 128 | 4608 | 1.769 | 289.50 | 2.999 | 42.68 |
| 512 | 128 | 5120 | 1.845 | 277.48 | 3.102 | 41.26 |
| 512 | 128 | 5632 | 1.893 | 270.46 | 3.219 | 39.76 |
| 512 | 128 | 6144 | 1.953 | 262.20 | 3.348 | 38.23 |
| 512 | 128 | 6656 | 2.018 | 253.71 | 3.474 | 36.84 |
| 512 | 128 | 7168 | 2.078 | 246.34 | 3.589 | 35.66 |
| 512 | 128 | 7680 | 2.140 | 239.22 | 3.717 | 34.43 |
| 512 | 128 | 8192 | 2.196 | 233.15 | 3.854 | 33.21 |
JSONL output
Pass --output-format jsonl to output JSONL instead of Markdown, á la
{"n_kv_max": 8704, "n_batch": 2048, "n_ubatch": 512, "flash_attn": 0, "n_gpu_layers": -1, "n_threads": 32, "n_threads_batch": 32, "pp": 512, "tg": 128, "n_kv": 0, "t_pp": 1.093814, "speed_pp": 468.086884, "t_tg": 1.780312, "speed_tg": 71.897514 }
{"n_kv_max": 8704, "n_batch": 2048, "n_ubatch": 512, "flash_attn": 0, "n_gpu_layers": -1, "n_threads": 32, "n_threads_batch": 32, "pp": 512, "tg": 128, "n_kv": 512, "t_pp": 1.169302, "speed_pp": 437.868073, "t_tg": 1.897474, "speed_tg": 67.458099 }
{"n_kv_max": 8704, "n_batch": 2048, "n_ubatch": 512, "flash_attn": 0, "n_gpu_layers": -1, "n_threads": 32, "n_threads_batch": 32, "pp": 512, "tg": 128, "n_kv": 1024, "t_pp": 1.183700, "speed_pp": 432.542053, "t_tg": 2.059179, "speed_tg": 62.160694 }
{"n_kv_max": 8704, "n_batch": 2048, "n_ubatch": 512, "flash_attn": 0, "n_gpu_layers": -1, "n_threads": 32, "n_threads_batch": 32, "pp": 512, "tg": 128, "n_kv": 1536, "t_pp": 1.428625, "speed_pp": 358.386566, "t_tg": 2.160639, "speed_tg": 59.241734 }
{"n_kv_max": 8704, "n_batch": 2048, "n_ubatch": 512, "flash_attn": 0, "n_gpu_layers": -1, "n_threads": 32, "n_threads_batch": 32, "pp": 512, "tg": 128, "n_kv": 2048, "t_pp": 1.360647, "speed_pp": 376.291595, "t_tg": 2.274003, "speed_tg": 56.288403 }