1163af96cf
get_mmq_x_max_host() returned 128 for NVIDIA Volta when built with
GGML_CUDA_FORCE_MMQ, but get_mmq_x_max_device() caps at
MMQ_DP4A_MAX_BATCH_SIZE (= 64) for the same configuration because the
wider mmq_x device kernels are not instantiated on sm_70. The host
dispatcher selects an mmq_x in [8, mmq_x_max] and instantiates the
matching template; any pick > 64 hits NO_DEVICE_CODE at runtime and
the kernel returns without writing output, producing zero-valued
activations and incoherent generation.
Repro on Tesla V100 (sm_70) with a Q6_K Qwen3.5-MoE model:
cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FORCE_MMQ=ON \
-DGGML_CUDA_NO_PEER_COPY=ON -DCMAKE_CUDA_ARCHITECTURES=70
./llama-cli -m model.gguf -ngl 99 -p "<~200-token prompt>"
# stderr: mmq.cuh:3941: ERROR: CUDA kernel mul_mat_q has no
# device code compatible with CUDA arch 700.
Cap the host bound at MMQ_DP4A_MAX_BATCH_SIZE on Volta regardless of
GGML_CUDA_FORCE_MMQ; this matches the device template availability and
yields correct output. Measured on a single V100-SXM2:
before patch: PP unusable (NO_DEVICE_CODE), TG ~100 t/s
after patch: PP ~630 t/s, TG ~102 t/s
The change is a no-op on architectures that have int8 MMA (Turing+),
on AMD/HIP, and on builds without GGML_CUDA_FORCE_MMQ.
|
||
|---|---|---|
| .. | ||
| cmake | ||
| include | ||
| src | ||
| .gitignore | ||
| CMakeLists.txt | ||