nicolas
/
ik_llama_opt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ik_llama_opt/ggml/src/ggml-cuda
History
Kawrakow ac4ec50f03 CUDA: muh faster prompt processing for MoE models and small u-batch sizes (#728) 
* WIP: adding mainline mmq_id implementation

* This seems to work

* Now also -fmoe works

* WIP

* WIP

* WIP

* This works for mainline supported quants

* mmq_id: add iq2_k, iq2_k_r4

* mmiq_id: don't assume row size is multiple of type size (per row scales)

* mmiq_id: don't assume row size is multiple of type size

* mmq_id: add iq2_ks

So we are sure it works with per row scales

* mmq_id: add iq2_kl

* mmq_id: add iq3_ks

* mmq_id: adding iq3_k, iq3_k_r4

* mmq_id: add iq4_kss, iq4_ks, iq4_ks_r4

* mmq_id: adding iq4_k, iq4_k_r4

* mmq_id: adding iq5_ks, iq5_ks_r4

* mmq_id: adding iq5_k, iq5_k_r4, q6_0

* mmq_id: adding iq6_k

* mmq_id: add iq1_s_r4

* mmq_id: adding iq1_kt, iq2_kt

* mmq_id: add iq3_kt, iq4_kt

* Add CUDA fp8 header

---------

Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
 		2025-08-26 13:30:35 +03:00
..
template-instances CUDA: muh faster prompt processing for MoE models and small u-batch sizes (#728) 2025-08-26 13:30:35 +03:00
vendors CUDA: muh faster prompt processing for MoE models and small u-batch sizes (#728) 2025-08-26 13:30:35 +03:00
acc.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
acc.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
add-id.cu Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
add-id.cuh Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
arange.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
arange.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
argsort.cu Add copyright notices (#317) 2025-04-07 10:43:26 +02:00
argsort.cuh Add copyright notices (#317) 2025-04-07 10:43:26 +02:00
binbcast.cu Add copyright notices (#317) 2025-04-07 10:43:26 +02:00
binbcast.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
clamp.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
clamp.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
common.cuh Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
concat.cu Add copyright notices (#317) 2025-04-07 10:43:26 +02:00
concat.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
conv-transpose-1d.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
conv-transpose-1d.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
convert.cu MXFP4 (#682) 2025-08-09 08:40:18 +03:00
convert.cuh Add copyright notices (#317) 2025-04-07 10:43:26 +02:00
cp-async.cuh Faster DeepSeek FA on CUDA (#408) 2025-05-12 07:49:00 +03:00
cpy-utils.cuh Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
cpy.cu remove curious assertions (#705) 2025-08-19 14:41:29 +03:00
cpy.cuh Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
dequantize.cuh Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
diagmask.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
diagmask.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
dmmv.cu Trellis quants with CPU inference (#441) 2025-05-23 09:17:52 +03:00
dmmv.cuh Merge mainline - Aug 12 2024 (#17) 2024-08-12 15:14:32 +02:00
fattn-common.cuh Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
fattn-mma-f16.cuh Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
fattn-new-mma.cu Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
fattn-new-mma.cuh FlashMLA-3 for DeepSeek models on CUDA (#386) 2025-05-07 17:38:22 +03:00
fattn-tile-f16.cu Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
fattn-tile-f16.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
fattn-tile-f32.cu Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
fattn-tile-f32.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
fattn-vec-f16.cuh Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
fattn-vec-f32.cuh Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
fattn-wmma-f16.cuh Fix GLM-4.5 attention (#700) 2025-08-17 14:31:03 +03:00
fattn.cu Fix GLM-4.5 attention (#700) 2025-08-17 14:31:03 +03:00
fattn.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
getrows.cu Add copyright notices (#317) 2025-04-07 10:43:26 +02:00
getrows.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
graph.cuh Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
im2col.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
im2col.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
iqk_cuda_common.h CUDA: faster IQ2_K, IQ2_KS, IQ2_K_R4 (#716) 2025-08-22 07:25:35 +03:00
iqk_mmvq.cu CUDA: faster IQ2_K, IQ2_KS, IQ2_K_R4 (#716) 2025-08-22 07:25:35 +03:00
iqk_mmvq.cuh Adding IQ1_KT - 1.75 bpw SOTA quants (#616) 2025-07-20 10:05:23 +02:00
mma.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
mma_new.cuh CUDA: faster FA TG for GQA models (#370) 2025-05-04 09:17:44 +03:00
mmq.cu CUDA: faster IQ2_K, IQ2_KS, IQ2_K_R4 (#716) 2025-08-22 07:25:35 +03:00
mmq.cuh CUDA: faster IQ2_K, IQ2_KS, IQ2_K_R4 (#716) 2025-08-22 07:25:35 +03:00
mmq_id.cu CUDA: muh faster prompt processing for MoE models and small u-batch sizes (#728) 2025-08-26 13:30:35 +03:00
mmq_id.cuh CUDA: muh faster prompt processing for MoE models and small u-batch sizes (#728) 2025-08-26 13:30:35 +03:00
mmq_id_common.cuh CUDA: muh faster prompt processing for MoE models and small u-batch sizes (#728) 2025-08-26 13:30:35 +03:00
mmvq.cu MXFP4 (#682) 2025-08-09 08:40:18 +03:00
mmvq.cuh Trellis quants with CPU inference (#441) 2025-05-23 09:17:52 +03:00
norm.cu cuda: non-contiguous rms norm (#190) 2025-02-07 08:33:42 +02:00
norm.cuh Adding fused rms_norm (#42) 2024-09-08 10:19:21 +03:00
pad.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
pad.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
pool2d.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
pool2d.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
quantize.cu CUDA: small PP performance improvement for MoE models (#589) 2025-07-07 07:23:12 +02:00
quantize.cuh CUDA: small PP performance improvement for MoE models (#589) 2025-07-07 07:23:12 +02:00
quantize_id.cu CUDA: muh faster prompt processing for MoE models and small u-batch sizes (#728) 2025-08-26 13:30:35 +03:00
quantize_id.cuh CUDA: muh faster prompt processing for MoE models and small u-batch sizes (#728) 2025-08-26 13:30:35 +03:00
rope.cu Add copyright notices (#317) 2025-04-07 10:43:26 +02:00
rope.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
scale.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
scale.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
softcap.cu Add copyright notices (#317) 2025-04-07 10:43:26 +02:00
softcap.cuh Add copyright notices (#317) 2025-04-07 10:43:26 +02:00
softmax.cu Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
softmax.cuh Add copyright notices (#317) 2025-04-07 10:43:26 +02:00
sumrows.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
sumrows.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
tsembd.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
tsembd.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
unary.cu Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
unary.cuh Enable CUDA graphs for MoE models + GPT-OSS support (#689) 2025-08-15 09:18:07 +03:00
upscale.cu Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
upscale.cuh Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
vecdotq.cuh CUDA: faster prompt processing for 4-bit quants (#713) 2025-08-21 15:57:35 +03:00