nicolas
/
ik_llama_opt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ik_llama_opt/ggml/src/ggml-cuda
History
Alex ab6d816818
CUDA DSA: fix v_offset for quantized K/V caches (#2387) 
v_offset is used as a column index into the dequantized f16 buffer, but was
computed as a half-pointer difference. That is only correct for f16 K/V. With
q8_0 (34 bytes per 32 elements) it yields the wrong column, so the sparse
attention path reads V from the wrong positions and generation degenerates.
 		2026-08-31 18:35:17 +02:00
..
template-instances cuda: fix iq3_k value table selection in MMVQ (#2364) 2026-08-27 08:40:14 +02:00
vendors cuda : repair the HIP build, and validate IQ4_KS and IQ4_KT on RDNA3 (#2339) 2026-08-25 08:50:44 +02:00
acc.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
acc.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
add-id.cu Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
add-id.cuh Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
arange.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
arange.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
argmax.cu Adding Ling/Ring (a.k.a., Bailing-MoE2) support (#833) 2025-10-15 14:20:40 +03:00
argmax.cuh Adding Ling/Ring (a.k.a., Bailing-MoE2) support (#833) 2025-10-15 14:20:40 +03:00
argsort.cu cuda : repair the HIP build, and validate IQ4_KS and IQ4_KT on RDNA3 (#2339) 2026-08-25 08:50:44 +02:00
argsort.cuh Fused indexer top_k (CUDA) (#2103) 2026-07-12 08:08:04 +03:00
binbcast.cu Add missing contiguity check (#2080) 2026-07-06 17:55:14 +02:00
binbcast.cuh WIP: Qwen3Next (#1266) 2026-02-16 06:50:28 +01:00
blend.cu deepseek2 : GLM-DSA sparse attention (lightning indexer), --dsa off by default (#2045) 2026-07-02 09:36:49 +02:00
blend.cuh deepseek2 : GLM-DSA sparse attention (lightning indexer), --dsa off by default (#2045) 2026-07-02 09:36:49 +02:00
clamp.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
clamp.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
common.cuh cuda : repair the HIP build, and validate IQ4_KS and IQ4_KT on RDNA3 (#2339) 2026-08-25 08:50:44 +02:00
concat.cu Revert CUDA concat change in #2179 (#2200) 2026-07-28 12:22:55 +03:00
concat.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
conv-transpose-1d.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
conv-transpose-1d.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
conv2d-dw.cu Port mdmd from mainline + Qwen2/2.5-VL support (#798) 2025-09-27 08:45:29 +02:00
conv2d-dw.cuh Port mdmd from mainline + Qwen2/2.5-VL support (#798) 2025-09-27 08:45:29 +02:00
conv2d.cu Chores : Typos fixing round 3 (project wide, ggml dir included, comments and user facing msg only) (#2249) 2026-08-04 07:15:28 +03:00
conv2d.cuh Port mdmd from mainline + Qwen2/2.5-VL support (#798) 2025-09-27 08:45:29 +02:00
convert.cu Fix non-contiguous batched cuBLAS (#1178) 2026-01-22 12:25:05 +02:00
convert.cuh Fix non-contiguous batched cuBLAS (#1178) 2026-01-22 12:25:05 +02:00
cp-async.cuh Faster DeepSeek FA on CUDA (#408) 2025-05-12 07:49:00 +03:00
cpy-utils.cuh Fix minor CUDA discrepencies (#2005) 2026-06-23 09:37:48 +02:00
cpy.cu Fix massive inefficiency in CUDA Q->f32/f16 and f32/f16->Q copies (#2279) 2026-08-08 17:26:59 +03:00
cpy.cuh DeepSeek TG optimizations for TG (#928) 2025-11-10 09:52:07 +02:00
cumsum.cu WIP: Qwen3Next (#1266) 2026-02-16 06:50:28 +01:00
cumsum.cuh WIP: Qwen3Next (#1266) 2026-02-16 06:50:28 +01:00
delta-net.cu ggml : fuse the delta-net recurrent state copy into the op (#2251) 2026-08-04 19:27:27 +03:00
delta-net.cuh Fused delta-net (#1315) 2026-02-25 14:12:48 +01:00
dequantize.cuh Enable iq4_nl KV cache on CUDA (#1006) 2025-11-24 09:41:19 +01:00
diagmask.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
diagmask.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
dmmv.cu Fix minor CUDA discrepancies (part 2) (#2015) 2026-06-23 14:03:22 +02:00
dmmv.cuh Merge mainline - Aug 12 2024 (#17) 2024-08-12 15:14:32 +02:00
ds4_comp.cu DS4 optimizations (#2169) 2026-07-25 08:52:38 +03:00
ds4_comp.cuh DS4 optimizations (#2169) 2026-07-25 08:52:38 +03:00
dsa_attn.cu CUDA DSA: fix v_offset for quantized K/V caches (#2387) 2026-08-31 18:35:17 +02:00
dsa_attn.cuh GLM-DSA: much better PP long context performance (CUDA) (#2109) 2026-07-12 19:33:33 +03:00
fattn-common.cuh Apply offfset to KQ_max in CUDA flash attention (#1196) 2026-01-29 07:27:53 +02:00
fattn-mma-f16-interface.cuh CUDA: corectly detect if flash attention is supported (#875) 2025-10-29 13:56:16 +02:00
fattn-mma-f16.cu Adding Muse-Glimmer support (#2293) 2026-08-12 15:53:54 +02:00
fattn-mma-f16.cuh Chores : Typos fixing round 3 (project wide, ggml dir included, comments and user facing msg only) (#2249) 2026-08-04 07:15:28 +03:00
fattn-new-mma.cu CUDA: handle GQA = 12 for head size = 256 via new MMA (#2372) 2026-08-28 18:10:50 +02:00
fattn-new-mma.cuh FlashMLA-3 for DeepSeek models on CUDA (#386) 2025-05-07 17:38:22 +03:00
fattn-tile-f16.cu CUDA: fix flash attention for gpt-oss (SWA + attention sinks) on the tile kernels (no-tensor-core GPUs) (#2087) 2026-07-06 17:59:29 +02:00
fattn-tile-f16.cuh CUDA: corectly detect if flash attention is supported (#875) 2025-10-29 13:56:16 +02:00
fattn-tile-f32.cu P100 tile-f32 exact-retile: half2 K/V smem staging (2-blocks/SM + leaner inner loop) (#2142) 2026-07-17 17:40:41 +03:00
fattn-tile-f32.cuh CUDA: corectly detect if flash attention is supported (#875) 2025-10-29 13:56:16 +02:00
fattn-vec-common.cuh Chores : Typos fixing round 3 (project wide, ggml dir included, comments and user facing msg only) (#2249) 2026-08-04 07:15:28 +03:00
fattn-vec-f16-interface.cuh CUDA: corectly detect if flash attention is supported (#875) 2025-10-29 13:56:16 +02:00
fattn-vec-f16.cu cuda : repair the HIP build, and validate IQ4_KS and IQ4_KT on RDNA3 (#2339) 2026-08-25 08:50:44 +02:00
fattn-vec-f16.cuh cuda : fix MLA flash-attn vec decode for asymmetric K/V head sizes (#2031) 2026-06-25 08:56:17 +02:00
fattn-vec-f32-interface.cuh CUDA: corectly detect if flash attention is supported (#875) 2025-10-29 13:56:16 +02:00
fattn-vec-f32.cu CUDA: MLA flash-attention decode on Pascal (vec_f32 K=576/V=512), incl. Q8_0 KV (#2079) 2026-07-06 17:58:50 +02:00
fattn-vec-f32.cuh cuda : repair the HIP build, and validate IQ4_KS and IQ4_KT on RDNA3 (#2339) 2026-08-25 08:50:44 +02:00
fattn-wmma-f16-interface.cuh CUDA: corectly detect if flash attention is supported (#875) 2025-10-29 13:56:16 +02:00
fattn-wmma-f16.cu CUDA: corectly detect if flash attention is supported (#875) 2025-10-29 13:56:16 +02:00
fattn-wmma-f16.cuh Fix GLM-4.5 attention (#700) 2025-08-17 14:31:03 +03:00
fattn.cu CUDA: handle GQA = 12 for head size = 256 via new MMA (#2372) 2026-08-28 18:10:50 +02:00
fattn.cuh CUDA: corectly detect if flash attention is supported (#875) 2025-10-29 13:56:16 +02:00
fill.cu WIP: Qwen3Next (#1266) 2026-02-16 06:50:28 +01:00
fill.cuh WIP: Qwen3Next (#1266) 2026-02-16 06:50:28 +01:00
getrows.cu Fix grid overflow in get_rows on CUDA (#2367) 2026-08-27 15:40:41 +02:00
getrows.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
graph.cuh CUDA graphs improvements (#2316) 2026-08-24 09:51:59 +02:00
hadamard.cu MLA TP -khad: ggml_dequant_hadamard fused op + wv_b/wk_b_pp Hadamard fold (#1852) 2026-05-21 07:29:15 +03:00
hadamard.cuh K-cache Hadamard transforms (CUDA) (#1034) 2025-12-04 18:46:22 +01:00
im2col.cu Fix gemma3 vision (#803) 2025-09-27 11:15:32 +02:00
im2col.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
indexer_topk.cu Actually fix quantized indexer cache on CUDA (#2286) 2026-08-10 18:46:02 +02:00
indexer_topk.cuh DS4: slowly approaching a meaningful performance (#2165) 2026-07-22 17:18:57 +03:00
iqk_cuda_common.h CUDA: faster IQ2_K, IQ2_KS, IQ2_K_R4 (#716) 2025-08-22 07:25:35 +03:00
iqk_mmvq.cu CUDA: fuse ffn_up*unary_op(ffn_gate) for MMVQ (V2) (#864) 2025-10-26 17:08:50 +02:00
iqk_mmvq.cuh CUDA: fuse ffn_up*unary_op(ffn_gate) for MMVQ (V2) (#864) 2025-10-26 17:08:50 +02:00
iqk_mmvq_templates.cuh cuda : repair the HIP build, and validate IQ4_KS and IQ4_KT on RDNA3 (#2339) 2026-08-25 08:50:44 +02:00
kda.cu model: Ling-3.0 (bailingmoe3) runtime support (#2295) 2026-08-13 08:02:50 +02:00
kda.cuh model: Ling-3.0 (bailingmoe3) runtime support (#2295) 2026-08-13 08:02:50 +02:00
latent_attn.cu openpangu: fused latent attention op (GGML_OP_LATENT_ATTN) (#2168) 2026-07-23 14:50:45 +03:00
latent_attn.cuh openpangu: fused latent attention op (GGML_OP_LATENT_ATTN) (#2168) 2026-07-23 14:50:45 +03:00
mma.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
mma_new.cuh CUDA: faster FA TG for GQA models (#370) 2025-05-04 09:17:44 +03:00
mmq.cu Compiler warning 2026-07-13 10:34:26 +00:00
mmq.cuh Fused indexer top_k (CUDA) (#2103) 2026-07-12 08:08:04 +03:00
mmq_id.cu fix: use mmq for volta quantized matmuls (#1785) 2026-05-15 08:11:49 +03:00
mmq_id.cuh Revert "CUDA: prompt processing optimizations for MoE models (#739)" (#748) 2025-09-02 06:55:48 +02:00
mmq_id_common.cuh Chores : Typos fixing round 3 (project wide, ggml dir included, comments and user facing msg only) (#2249) 2026-08-04 07:15:28 +03:00
mmvq-args.h Step-3.5-Flash support (#1231) 2026-02-05 08:13:22 +02:00
mmvq-templates.cuh Split mode graph for MiniMax-M3 2026-06-15 08:41:34 +00:00
mmvq.cu Split mode graph for MiniMax-M3 2026-06-15 08:41:34 +00:00
mmvq.cuh Step-3.5-Flash support (#1231) 2026-02-05 08:13:22 +02:00
multiadd.cu Fuse some ops for Gemma4-MoE (#1610) 2026-04-11 08:11:54 +02:00
multiadd.cuh Fused mul + multi_add op (#858) 2025-10-24 07:40:35 +03:00
norm.cu CUDA: fuse rms -> add -> rms (#2297) 2026-08-13 15:24:45 +02:00
norm.cuh CUDA: fuse rms -> add -> rms (#2297) 2026-08-13 15:24:45 +02:00
pad.cu More Qwen3-Next optimizations (#1277) 2026-02-17 16:03:51 +01:00
pad.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
pool2d.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
pool2d.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
quantize.cu Chores : Typos fixing round 3 (project wide, ggml dir included, comments and user facing msg only) (#2249) 2026-08-04 07:15:28 +03:00
quantize.cuh CUDA MoE improvements (#923) 2025-11-09 11:34:33 +02:00
quantize_id.cu Chores : Typos fixing round 3 (project wide, ggml dir included, comments and user facing msg only) (#2249) 2026-08-04 07:15:28 +03:00
quantize_id.cuh CUDA: muh faster prompt processing for MoE models and small u-batch sizes (#728) 2025-08-26 13:30:35 +03:00
reduce.cu Split mode graph for dense Gemma4 assistant (#2022) 2026-06-24 18:29:32 +02:00
reduce.cuh Graph parallel: the next generation (#1080) 2025-12-24 08:31:48 +01:00
rope.cu DS4: streamline RoPE (#2198) 2026-07-29 07:36:53 +03:00
rope.cuh Fuse Q and K RoPE (#980) 2025-11-19 09:08:42 +01:00
scale.cu Chores : Typos fixing round 3 (project wide, ggml dir included, comments and user facing msg only) (#2249) 2026-08-04 07:15:28 +03:00
scale.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
set-rows.cu Adding Ling/Ring (a.k.a., Bailing-MoE2) support (#833) 2025-10-15 14:20:40 +03:00
set-rows.cuh Adding Ling/Ring (a.k.a., Bailing-MoE2) support (#833) 2025-10-15 14:20:40 +03:00
sinkhorn.cu DS4: slowly approaching a meaningful performance (#2165) 2026-07-22 17:18:57 +03:00
sinkhorn.cuh DS4: slowly approaching a meaningful performance (#2165) 2026-07-22 17:18:57 +03:00
softcap.cu Add copyright notices (#317) 2025-04-07 10:43:26 +02:00
softcap.cuh Add copyright notices (#317) 2025-04-07 10:43:26 +02:00
softmax.cu Chores : Typos fixing round 3 (project wide, ggml dir included, comments and user facing msg only) (#2249) 2026-08-04 07:15:28 +03:00
softmax.cuh Add copyright notices (#317) 2025-04-07 10:43:26 +02:00
solve_tri.cu cuda : repair the HIP build, and validate IQ4_KS and IQ4_KT on RDNA3 (#2339) 2026-08-25 08:50:44 +02:00
solve_tri.cuh WIP: Qwen3Next (#1266) 2026-02-16 06:50:28 +01:00
ssm-conv.cu MTP: faster recurrent state restore (#1791) 2026-05-13 11:00:24 +03:00
ssm-conv.cuh WIP: Qwen3Next (#1266) 2026-02-16 06:50:28 +01:00
sumrows.cu New op: ggml_sum_rows_ext (#2132) 2026-07-15 16:21:17 +03:00
sumrows.cuh WIP: Qwen3Next (#1266) 2026-02-16 06:50:28 +01:00
topk-moe.cu Fuse experts bias in top_k_moe kernel (#1170) 2026-01-20 15:38:51 +02:00
topk-moe.cuh Fuse experts bias in top_k_moe kernel (#1170) 2026-01-20 15:38:51 +02:00
tri.cu WIP: Qwen3Next (#1266) 2026-02-16 06:50:28 +01:00
tri.cuh WIP: Qwen3Next (#1266) 2026-02-16 06:50:28 +01:00
tsembd.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
tsembd.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
unary.cu Merged up/gate for DS4 (#2257) 2026-08-07 07:51:58 +03:00
unary.cuh DS4: slowly approaching a meaningful performance (#2165) 2026-07-22 17:18:57 +03:00
upscale.cu Feat - add kimi 2.5 Vision (#1280) 2026-02-19 08:15:03 +01:00
upscale.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
vecdotq.cuh cuda : repair the HIP build, and validate IQ4_KS and IQ4_KT on RDNA3 (#2339) 2026-08-25 08:50:44 +02:00