ik_llama_opt/ggml/src
Gearstickle 4fbd0c441b
fa: preserve early-termination, fix multi-slot correctness via union of masks (#1880)
* fa: fix FlashQKV early-termination causing S=0 assertion with --parallel N>1

The backward-scan optimization in compute_helper/compute_helper_q checks
only one mask position per k_step block on the last query row (q_step-1)
to find where valid KV entries end. When q_step > 1 and different query
rows have non-overlapping valid KV regions (multi-slot / --parallel N>1),
the scan on the last row's mask can miss blocks that contain valid entries
for earlier rows. This causes those rows to accumulate S=0, triggering
the GGML_ASSERT(S > 0) in normalize_and_store_1row.

Fix: remove the early-termination scan at all 4 sites and iterate all
nk1/k_step blocks unconditionally. The mask already handles correctness:
fully-masked blocks produce smax=-inf and skip V accumulation, so the
performance cost is minimal for TG (small nq1) and acceptable for PP.

Fixes #809

* fa: refactor multi-slot mask fix into mask_effective_nk1() helper

Replace 4× inlined early-termination scans with a shared helper that
computes the effective K boundary by scanning ALL query mask rows
(union-of-masks). This is the minimal fix for multi-slot parallel
inference where different slots have different sequence lengths.

The helper returns the k_step-aligned boundary covering the longest
active sequence across all rows, preserving single-slot performance
(single row = same boundary as before).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Turbomen008 <Turbomen008@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-26 16:16:49 +03:00
..
cmake Merge vulkan code from mainline up to commit of 6/28/2025 (#563) 2025-07-02 08:49:42 +02:00
ggml-cann Merge mainline - Aug 12 2024 (#17) 2024-08-12 15:14:32 +02:00
ggml-cuda MLA TP -khad: ggml_dequant_hadamard fused op + wv_b/wk_b_pp Hadamard fold (#1852) 2026-05-21 07:29:15 +03:00
ggml-sycl Merge mainline - Aug 12 2024 (#17) 2024-08-12 15:14:32 +02:00
iqk fa: preserve early-termination, fix multi-slot correctness via union of masks (#1880) 2026-05-26 16:16:49 +03:00
kompute@4565194ed7 Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
kompute-shaders Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
llamafile Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
vulkan-shaders Port of Qwen3-VL support from mainline (#883) 2025-11-04 19:20:54 +02:00
CMakeLists.txt Quantization options (#1677) 2026-04-23 09:05:39 +02:00
ggml-aarch64.c Merge mainline - Aug 12 2024 (#17) 2024-08-12 15:14:32 +02:00
ggml-aarch64.h Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
ggml-alloc.c Support for Qwen 3.5 MTP (dense models only) (#1698) 2026-04-28 07:47:50 +02:00
ggml-backend-impl.h Merge vulkan code from mainline up to commit of 6/28/2025 (#563) 2025-07-02 08:49:42 +02:00
ggml-backend.cpp Fix crash with split mode graph and partial offload (#1857) 2026-05-21 13:36:01 +03:00
ggml-blas.cpp Merge mainline - Aug 12 2024 (#17) 2024-08-12 15:14:32 +02:00
ggml-cann.cpp Merge vulkan code from mainline up to commit of 6/28/2025 (#563) 2025-07-02 08:49:42 +02:00
ggml-common.h Bonsai support (AVX2, generic) (#1570) 2026-04-02 16:54:08 +02:00
ggml-cuda.cu Fix split mode graph for Qwen35-MoE + MTP (#1861) 2026-05-22 09:23:53 +03:00
ggml-impl.h MXFP4 (#682) 2025-08-09 08:40:18 +03:00
ggml-kompute.cpp Merge vulkan code from mainline up to commit of 6/28/2025 (#563) 2025-07-02 08:49:42 +02:00
ggml-metal.m Fix clang warnings on macOS (#1354) 2026-03-03 16:27:16 +01:00
ggml-metal.metal MXFP4 (#682) 2025-08-09 08:40:18 +03:00
ggml-quants.c Use AVX version VNNI intrinsic when AVX512VNNI not available. (#1748) 2026-05-09 09:02:06 +03:00
ggml-quants.h Quantization options (#1677) 2026-04-23 09:05:39 +02:00
ggml-rpc.cpp server: improve speed of speculative decoding (#1119) 2026-01-10 08:01:22 +02:00
ggml-sycl.cpp Merge vulkan code from mainline up to commit of 6/28/2025 (#563) 2025-07-02 08:49:42 +02:00
ggml-vulkan.cpp Port of Qwen3-VL support from mainline (#883) 2025-11-04 19:20:54 +02:00
ggml.c MLA TP -khad: ggml_dequant_hadamard fused op + wv_b/wk_b_pp Hadamard fold (#1852) 2026-05-21 07:29:15 +03:00