72201359dd
On GPUs without FP16 tensor cores (Pascal / sm_60, e.g. Tesla P100) MLA flash-attention decode falls back to the CPU. The !fp16_mma_available path routes decode to the f16 vector kernel, whose is_supported check requires K == V head sizes; MLA's absorbed head sizes are 576/512 (asymmetric), so it is rejected and attention runs on the CPU. With --cpu-moe that recomputes the full MLA attention on the CPU every decoded token, which dominates decode at long context. Route Pascal MLA decode (Q->ne[1] <= 8 && K == 576 && V == 512) to the f32 vector kernel and enable that kernel for the 576/512 case, including Q8_0 KV. Scope: decode only (batch <= 8). Prefill (batch > 8) and -fa 0 are untouched; tensor-core GPUs never reach this branch. Aligned head sizes are byte-identical (the asymmetric/Q8_0 work folds to a no-op at compile time), so no other model or configuration is affected. - fattn.cu: route 576/512 decode to vec_f32 in the !fp16_mma dispatch and its is_supported mirror. - fattn-vec-f32.cu/.cuh: accept + instantiate 576/512 (F16 and Q8_0); fix latent issues exposed by the first asymmetric/large-head use (KQ-row granularity uses FATTN_KQ_STRIDE not Dv; guard the dst store to tid < Dv; guard the softmax exp on the KV tail; size Q_i32 by ceil; only convert K/V to F16 when the type is F16). All are no-ops for the previously-exercised symmetric cases. - fattn-vec-f32.cuh / fattn-vec-common.cuh: guard the Q8_0 ragged tail (Dk=576 is 144 int32 lanes = 4.5 warps) with the ragged-dim idiom; compile-time-constant for aligned head dims, so it folds away. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| cmake | ||
| include | ||
| src | ||
| .gitignore | ||
| CMakeLists.txt | ||