* cuda : add the missing compile definitions to the HIP build
GGML_CUDA_FUSION, GGML_CUDA_MIN_BATCH_OFFLOAD and GGML_CUDA_PEER_MAX_BATCH_SIZE
are used unconditionally in common.cuh but were only defined in the CUDA branch,
so every HIP translation unit failed to compile. GGML_CUDA_IQK_FORCE_BF16 and
GGML_CUDA_F16 are user facing options the HIP branch silently ignored.
Also define GGML_USE_HIP. 43 tests in the sources imported from upstream use that
spelling, this fork only defined the older GGML_USE_HIPBLAS, so all of them took
the NVIDIA branch. Without it mmq_id_common.cuh defines TURING_MMA_AVAILABLE,
AMPERE_MMA_AVAILABLE, CP_ASYNC_AVAILABLE and FP16_MMA_AVAILABLE, i.e. the inline
PTX paths, and mmq_id.cu and mmq-instance-q6_k_id.cu then fail to build on
mma_new.cuh:181.
* cuda : shim the warp sync primitives in the HIP vendor header
ROCm 6 and later provide __shfl_sync() and friends as templates that static_assert
on the width of the mask, since an AMD wave can be 64 lanes wide. Define
HIP_DISABLE_WARP_SYNC_BUILTINS so that the shims replace them instead of clashing
with them, and add the shims the header was missing.
This is what makes mmq_id_common.cuh compile, so it unblocks all 26 mmq-instance-*_id
translation units, i.e. the MoE mat-mat path for the iqk quants.
* cuda : update the HIP vendor header for ROCm 6 and 7
- map nv_bfloat16 and nv_bfloat162 onto the __hip_bfloat16 types. Nothing declared
them, so every translation unit that mentions bf16 failed, convert.cu included -
that is the dequantize path the iqk quants use for prompt processing.
- use the hipblasComputeType_t and hipDataType entry points from ROCm 6.5 onwards.
hipblasDatatype_t is deprecated there and no longer matches hipblasGemmEx().
- make cudaStreamWaitEvent object-like. As a 3 argument function-like macro it did
not expand at the 2 argument call sites in reduce.cu, and the unexpanded name was
then passed on to CUDA_CHECK.
- add the mappings for cudaOccupancyMaxActiveBlocksPerMultiprocessor, which 29 of
the 30 failing flash attention translation units needed, and for the entry points
used by dsa_attn.cu and solve_tri.cu. hipBLAS spells a half as an unsigned short,
so cublasHgemmStridedBatched goes through a small casting wrapper rather than a
plain rename, which would not compile at the dsa_attn.cu call site.
* cuda : fix the remaining HIP build errors
- argsort.cu declared the sort order inside #ifdef GGML_CUDA_USE_CUB and used it
outside, and indexer_topk.cu calls argsort_f32_i32_cuda_cub() unconditionally
while it is only defined when CUB is available. Both break any build without CUB,
which includes MUSA and CUDA older than 11.7, not just HIP.
- ggml_backend_cuda_invalidate_graphs() touched ctx->cuda_graphs, which only exists
under USE_CUDA_GRAPH. The function is exported and called from llama-reload.cpp,
so guard the body rather than the function.
- solve_tri.cu included <cublas_v2.h> directly. common.cuh already pulls in whichever
vendor header is right, so the include is removed - it was redundant on CUDA too.
- dsa_attn.cu passed a data type where the GEMM entry point wants a compute type.
hipblasGemmStridedBatchedEx() has no data type taking overload. cuBLAS does, but it
is deprecated: cublas_api.h migrates CUDA_R_32F to exactly CUBLAS_COMPUTE_32F unless
the handle is in CUBLAS_PEDANTIC_MATH, which nothing in this tree sets. So this also
moves the CUDA build onto the primary entry point and drops a cublasGetMathMode()
per call. ggml-cuda.cu already passes a compute type to the same function.
- two mmvq instances called __dp4a() directly instead of ggml_cuda_dp4a(), the
wrapper the rest of the backend uses. On CUDA the wrapper is __dp4a() for every
architecture that has it.
- cap the flash attention vec f32 kernel at 4 columns per block on HIP. Both
logit_softcap variants of the 8 column kernel in one module overflow the 16 bit
branch offset of the AMDGPU backend.
* cuda : build the missing template instances in the HIP build
The HIP source list had drifted from the CUDA one and left out three families of
template instances that the backend references unconditionally:
- mmvq-instance-*.cu, the only definition site for the iqk mat-vec entry points.
iqk_mmvq.cu calls mul_mat_vec_iq4_ks_q8_1_cuda() and mul_mat_vec_iq4_kt_q8_1_cuda()
and nothing defined them, so the library did not link.
- the fattn-vec instances for q8_0-iq4_nl, iq4_nl-iq4_nl, q6_0-q5_0 and q8_0-q6_0,
which fattn-vec-f16.cu and fattn-vec-f32.cu dispatch to in the default
configuration.
- fattn-mma-*.cu. The MMA kernels are never selected on AMD, new_mma_available()
requires an NVIDIA device, but fattn-mma-f16.cu still needs the symbols.
* cuda : recognise AMD GPUs in GGML_CUDA_CC_IS_NVIDIA
CC_OFFSET_AMD is 1000000 and CC_OFFSET_MTHREADS is 0x100000, i.e. 1048576, so the
whole AMD range sits below the Moore Threads offset and every AMD GPU tested as
NVIDIA. turing_mma_available() then returned true on RDNA, the host picked an MMQ
tile of 128 while get_mmq_x_max_device() caps at 64 on AMD, and mul_mat_q_id hit
its NO_DEVICE_CODE guard and wrote NaNs. MUL_MAT_ID on IQ4_KS and IQ4_KT failed
this way on gfx1101.
No effect on CUDA, where a compute capability is 100*major + 10*minor and is
always far below CC_OFFSET_AMD.
* cuda : use v_perm_b32 for the 4 bit table lookup on HIP
HIP implements __byte_perm() in software: it stores an 8 byte union and does four
dynamically indexed byte loads, which end up in scratch. get_int_from_table_16()
calls it eight times per 32 weights, so every quant with a value table was paying
for that, while the trellis types were not.
__builtin_amdgcn_perm() is v_perm_b32, one instruction, and does the same job.
Taken from ggml-org/llama.cpp, which already carries this path.
Token generation on a 7800 XT, pure quantized Qwen2.5-1.5B, tg128:
IQ4_KS 16.30 -> 255.88 t/s
IQ4_XS 17.28 -> 268.05 t/s
IQ4_KT 201.76 -> 195.68 t/s (no table, unchanged)
Perplexity is unchanged to every printed digit and still matches the CPU exactly.
The function is duplicated in vecdotq.cuh and iqk_mmvq_templates.cuh, so both
copies need it - the iqk mat-vec instances only see the latter.
* cuda : use the shared flash attention support check on HIP
supports_op() carried a hand-rolled head size test for HIP that predates the
shared check: it accepted head size 64 with an f16 K cache and head size 128,
and nothing else. Head size 256 was rejected outright, so Gemma-2 and every
other 256 wide model fell back to the CPU for attention even though the
instances are compiled. @hardWorker254 reported 256 working with
ROCm 7.2.4 for both the f16 and the q8_0 cache.
Rather than adding 256 to the list, drop the branch and call
ggml_cuda_fattn_is_supported() as every other backend path does. It already
handles AMD: for cc >= CC_OFFSET_AMD it defers to the vec f16 or vec f32
support predicate depending on precision, which is exactly what
ggml_cuda_flash_attn_ext() dispatches to on AMD, because fast_fp16_available()
is true across the whole AMD cc range. The two now cannot drift.
This also removes a latent abort. The hand-rolled test returned true for any
head size 128 case regardless of the K and V types, so a combination without a
compiled instance, q4_1/q4_1 in a default build, reached the dispatcher and hit
on_no_fattn_vec_case() -> GGML_ABORT instead of falling back to the CPU. The
shared predicate is derived from the instances the build actually contains, and
after the source list repair earlier in this series the HIP build compiles the
same set as the CUDA build.
Beyond head size 256 this also lets HIP claim the asymmetric 192/128 and
576/512 vec f32 paths under GGML_PREC_F32. Those are untested on AMD; they are
gated by the same predicate CUDA uses.
* cuda : test the V head size, not the KV head count, for 192/128 vec f16 FA
ggml_cuda_fattn_vec_f16_is_supported() gates the asymmetric Dk != Dv branch on
if (K->ne[0] != 192 || V->ne[2] != 128) return false;
but ne[2] on K and V is the number of KV heads, not a head size. The test was
meant to be V->ne[0], as the wmma predicate added in the same commit (0459f595)
already writes it:
if (K->ne[0] != V->ne[0]) return K->ne[0] == 192 && V->ne[0] == 128;
and as the f32 twin has written it since 72201359 reworked that branch for
576/512. Only the f16 copy was left behind.
The kernels are there: ggml_cuda_flash_attn_ext_vec_f16() dispatches
FATTN_VEC_F16_CASE_DKDV(192, 128, ...) for f16-f16 and q8_0-q8_0 in both the
default and the GGML_CUDA_FA_ALL_QUANTS configuration, and the corresponding
hs192 instances are in the source list either way. The predicate just never
reported them, so a 192/128 shape whose KV head count was not coincidentally
128 was declined and attention fell back to the CPU.
This belongs in this series because the previous commit is what makes the
predicate reachable on AMD: with supports_op() routing flash attention through
ggml_cuda_fattn_is_supported(), the cc >= CC_OFFSET_AMD branch selects this
predicate for the default precision at every batch size, matching what
ggml_cuda_flash_attn_ext() dispatches to there. Without the fix the HIP build
would trade one hardcoded head size restriction for another.
NVIDIA is unaffected either way. Volta and later route 192/128 through the mma
or wmma predicates, and on Pascal the Q->ne[1] <= 8 decode case is diverted to
vec f32 before this predicate is consulted.
* fix: wrong tensor index in BF16 fused RMS norm add path (norm.cu:1039)
The BF16 branch of ggml_cuda_op_fused_rms_rms_add used dst->src[2]->data
for the second weight pointer, but should have used dst->src[3]->data.
This caused reading float weights from the wrong bf16 input tensor.
The F32 and F16 branches both correctly reference src[3], and the
assertions at lines 1013-1015 confirm src[3] is the F32 weight tensor.
* fix: off-by-one bounds check in 7 dmmv kernels (row > nrows -> row >= nrows)
Seven K-quant dequantize_mul_mat_vec kernels used row > nrows for bounds
checking instead of row >= nrows. Since rows are 0-indexed (0..nrows-1),
the check missed the row == nrows case, allowing a potential out-of-bounds
memory write when grid dimensions produce exactly nrows.
The templated dequantize_mul_mat_vec<type> kernel at line 667 already used
the correct row >= nrows pattern.
* fix: typo in function name iqk_mul_mat_vec_q_kerne -> iqk_mul_mat_vec_q_kernel
Truncated function name in iqk_mmvq_templates.cuh was missing trailing 'l'.
* fix: print actual split_dim value in set_tensor error message (ggml-cuda.cu)
fprintf used extra->split_dim == 0 which evaluates to boolean 0 or 1
instead of the actual split dimension value. When this fatal error is
hit for an unsupported split_dim, the user could not diagnose which
value caused the problem.
* fix: wrong src index in gate bias stride for fused up-gate MoE path
ggml_cuda_add_id for the gate bias used dst->src[4]->nb[1] as the stride
argument instead of dst->src[5]->nb[1]. This was a copy-paste error from
the up-bias code (lines 3220-3224) where src[4] is correct. If src[4]
and src[5] have different strides, the bias addition produces incorrect
results.
* fix: wrong row count for gate projection MMQ in fused up-gate MoE path
ggml_cuda_op_mul_mat_q for the gate projection (src0_2) used
src0_1->ne[1] as row_high instead of src0_2->ne[1]. This copy-paste
error causes processing the wrong number of rows if the up and gate
projections have different row counts. The gemv path (line ~3563)
correctly used src0_2->ne[1].
* WIP
* This works but is slow
* Turn off the up / gate clamps for now
* OK we need the clamping
* Fuse the clamp (CUDA)
* Fuse the clamp (CPU)
* WIP
* Be able to use merged q, k, v
* Be able to use merged up/gate experts
* Fuse the clamp (CUDA mmvq)
* Remove most of split mode row
* WIP
* WIP: also allocate the KV cache using tensor split
* WIP: it runs with wrong result
But it also looks like the backend scheduler is not going to help:
* It copies mask and input positions to GPU 0
* => RoPE ops must run on GPU 0
* => To proceed attn evaluation, GPU 1 must wait for GPU 0 to finish its
entire attn calculation
* Same with FFN. The rms_norm gets scheduled on GPU 0. Hence, GPU 1 must
wait for GPU 0 to finish its entore FFN calculation before it can
start (as it needs to copy the result of rms_norm from GPU 0)
* => Seems useless without writing a bespoke TP scheduling
* WIP
* This works, but it is slow
* This is slightly better
the graph is still not being computed in parallel.
Why? Because the scheduler creates graph splits where the
result of the computation on one GPU becomes an input for the
other split. Hence, to trigger the computation on the second GPU
one needs to wait for the computation on the first GPU to finish,
even thiough the two can be done in parallel up to the sunchronization
point. So, all that is left to do is to trick the scheduler to create
to splits that can be done in parallel, and then have a graph split
where the results get combined.
* Playing games with the scheduler
This change tricks it into doing the right thing^TM.
Still quite a bit slower than split mode layer for the 8B LlaMA model.
But for the 70B LlaMA it now beats split mode layer for TG:
28 t/s vs 24.4 t/s. PP is 627 t/s vs 744 t/s.
In comparison, split mode "row" in mainline gets
484 t/s PP and 19.3 t/s TG.
* Fix attn split
Granularity for Wq, Wo is not just head size, but
head size * gqa_ratio.
Else the Wk, Wv tensors end up not being a multiple of the
head size when we divide the split determined by Wo with
the gqa_ratio.
* Show memory used per device
* Make it work with partial offload
but no tensor overrides yet, just ngl < num_layers.
* Allow for f16 source in fused_rms_norm
* This results in faster PP.
Now PP is faster than split mode layer for L3-70B.
* Rename split mode "row" to split mode "graph"
* Leave FFN partial results as f16
* WIP GLM4.5 - runs with wrong results
* WIP GLM4.5 - this works
PP is already better than split mode layer, but TG for zero context
is kind of low - 60 vs 92 t/s. TG becomes better than split mode layer
at around 20k tokens. PP at 26k tokens is 1.55X of sm layer.
* Work around compiler bug
It issues a warning that there is an extra semicolon outside of a function,
but there isn't. If I remove the anonymous namespace and turn the
functions inside into static, the warning disapears, so clearly
a compiler bug.
* Make graph reuse work with split mode graph
* Remove more split mode row remnants
* WIP tensor overrides
Runs with wrong results, don't see where the issue could be.
* This works but is slow
Still does not work for row-interleaved quants
* Slightly better
* Slightly better
* Row-interleaved quants work
* Better
* Minor
* Guarad against using split mode "graph" for unsupported models
* Guards against using merge_qkv with split mode "graph"
* WIP split mode attn
Works for LlaMA models, but not for GLM-4.5.
Doesn't seem to improve performance, so I guess no point in trying to
fix it.
* Split mode graph for qwen3moe
* Try to better distribute the splits
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
* Args for MMVQ functions
* WIP
* Fused ffn_up*unary_op(ffn_gate) for MMVQ (no bias)
We see nearly 2% TG speedup for Ling-mini-2.0 and
about 1% for DeepSeek-Lite.
* Fused ffn_up*unary_op(ffn_gate) for MMVQ (with bias)
* Fusing also for iqk/trellis/repacked quants
* Fusing mmvq also in non-MoE up+gate
* Fuse mul_mat_id and add_id into a single kernel for mmvq
* Also iqk quants
* Split mmvq.cu and iqk_mmvq.cu into separate template instances
* Put iqk mmvq implementations into template instances
* Somehow I forgot to change the ggml_type in the legacy template calls
* Add disagnostics
* Disable assert
* Fix TG fused up*nary(gate) when down cannot be fused
The wrong memory buffer got used in that case
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Guy Barel
Nexes the Elder
Kawrakow