* IQK AVX2: Replace MM256_SET_M128I(x, x) identity broadcasts with _mm256_broadcastsi128_si256
## Summary
Replace all 132 instances of the identity-broadcast pattern
`MM256_SET_M128I(x, x)` with `_mm256_broadcastsi128_si256(x)` across
8 files in ggml/src/iqk/.
## Correctness
The transformation is bit-exact. The `MM256_SET_M128I(a, b)` macro
expands to:
_mm256_insertf128_si256(_mm256_castsi128_si256(b), a, 1)
which places `a` in the high 128-bit lane and `b` in the low lane.
When `a == b == x`, the result is x replicated to both lanes:
[x_lo: x_hi] = {x, x}
`_mm256_broadcastsi128_si256(x)` produces the identical register state:
it copies the 128-bit input to both lanes in a single micro-op.
Perplexity was verified identical on llama-3.2-1b-Q8_0 and
google-gemma-3-4b-Q4_0-IQ4_XS before and after the change.
The transformation is a pure intrinsic substitution with
zero semantic difference.
## Performance
### Micro-architecture analysis
The old pattern compiles to:
vinsertf128 ymm, ymm, xmm, 1 -- 3 uops, port 5, 3-cycle latency
The new pattern compiles to:
vbroadcasti128 ymm, xmm -- 1 uop, port 5, 1-cycle latency
Both instructions execute on port 5 (Intel), but vbroadcasti128:
- Uses 1/3 the dispatch slots (fewer pipeline stalls)
- Has 3x better latency (1 vs 3 cycles)
- Is not lane-crossing (no bypass delay between 128-bit halves)
### Measured results
#### Test 1: llama-3.2-1b-Q8_0, 2048 ctx, -b 128 -ub 128
Compiler | Metric | Before | After | Change
--------------|--------|---------|---------|-------
MSVC 19.44 | PP t/s | 596.89 | 604.57 | +1.29% (noise imo)
MSVC 19.44 | TG t/s | 55.13 | 55.72 | +1.07% (noise)
Clang 19.1.5 | PP t/s | 645.72 | 700.93 | +8.55% (systematic gain)
Clang 19.1.5 | TG t/s | 54.26 | 54.06 | -0.37% (noise)
#### Test 2: google-gemma-3-4b-Q4_0-IQ4_XS, 4096 ctx, -b 512
Compiler | Metric | Before | After | Change
--------------|----------|----------|----------|-------
Clang 19.1.5 | PP t/s | 262.40 | 314.79 | +19.96% (systematic gain)
Clang 19.1.5 | TG t/s | 30.78 | 32.32 | +5.00% (noise, TG oscilates between 31.5 and 33 t/s before / after)
Clang 19.1.5 | Total ms | 48881 | 44696 | -8.56%
Both tests ran on Intel Core Ultra 265K, 18 threads, flash_attn=1
The IQ4_XS result shows a dramatic PP improvement (+20%) because this
quantization format uses significantly more identity broadcasts in its
dequantization path (lookup-table expansion, scale duplication). The
compressed 4-bit representation requires more setup per block, making
the broadcast-to-256 step a measurable bottleneck that the 1-uop
vbroadcasti128 eliminates.
### Why these 132 instances matter
Every dequantization path (IQ2_XXS through IQ6_K, Q4_0 through Q8_1,
MXFP4) starts by broadcasting a 128-bit lookup table or scale vector
to 256 bits. These are in the inner loop of every quantization format's
dot-product kernel. Reducing each broadcast from 3 uops to 1 uop
cumulatively reduces port-5 pressure across the entire dequant
pipeline.
## Scope
This change touches only the identity-broadcast case (both MM256_SET_M128I
arguments are identical).
* IQK AVX2: Introduce MM256_SET1_M128I(x) wrapper for identity-broadcast pattern
Per Ikawrakow's review: replace direct _mm256_broadcastsi128_si256(x) with
a new macro MM256_SET1_M128I(x) wrapping the intrinsic, so that if the
broadcast turns out harmful on some CPU, only the macro definition needs
changing, not 132 call sites.
#define MM256_SET1_M128I(x) _mm256_broadcastsi128_si256(x)
The 132 identity-broadcast MM256_SET_M128I(x, x) call sites across 8 files
now use MM256_SET1_M128I(x) instead of the raw intrinsic.
* q8_k_r16: basics
* q8_k_r16: iq4_xs now uses q8_k_r16 on Zen4+
PP performance is about the same as using q8_k_r8 on the Ryzen-7950X,
so we expect nice gains on Zen5, and we don't need to wory about
using 2 different q8_k_r8 implementations for fancy SIMD.
* q8_k_r16: iq2_xxs now uses q8_k_r16 on Zen4+
* q8_k_r16: iq2_xs now uses q8_k_r16 on Zen4+
* q8_k_r16: iq2_s now uses q8_k_r16 on Zen4+
* q8_k_r16: iq3_xxs now uses q8_k_r16 on Zen4+
* q8_k_r16: iq3_s now uses q8_k_r16 on Zen4+
* q8_k_r16: iq1_s and iq1_m now uses q8_k_r16 on Zen4+
* q8_k_r16: q2_K and q3_K now uses q8_k_r16 on Zen4+
* q8_k_r16: iq2_ks and iq2_k now uses q8_k_r16 on Zen4+
* q8_k_r16: iq2_kl now uses q8_k_r16 on Zen4+
* q8_k_r16: iq3_ks and iq3_k now uses q8_k_r16 on Zen4+
* q8_k_r16: iq4_kss, iq4_ks, and iq4_k now use q8_k_r16 on Zen4+
* q8_k_r16: iq5_ks, iq5_k, and iq6_k now use q8_k_r16 on Zen4+
* Fix AVX2
* Just always set num_rows to 16
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
* q6_K dequantizing GEMM
* Much easier: just use different vec_dot types!
* WIP
* Finally q6_K x q8_2_x4 dot product works
* Very slightly better
* We don't need the changes in ggml.c
* Fix AVX2
* iq2_xs
* Fix AVX2
* iq2_s
* q3_K
* Fix q8_k_r8 on Zen4
* q3_K: repack to q8_k_r8 instead of q8_0_r8
With that we hit 360 t/s for LlaMA-3.1-8B on a Ryzen-7950X.
q8_k_r8 is 386 t/s, so for a batch size of 512 repacking costs
~7% of the time taken by the actual GEMM.
* q3_K: don't scale when all quants in a block are <= 127 when repacking
* iq2_s: repack to q8_k_r8 instead of q8_0_r8
* iq2_xs: rapck to q8_k_r8
* WIP
* iq2_xs: repack to q8_k_r8
* iq3_xxs: repack to q8_k_r8
* iq3_s: use q8_k_r8
* iq1_s: repack to q8_k_r8
* iq1_m: repack to q8_k_r8
* iq1_m: slightly faster
* Slightly faster
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
* Refactor iqk: WIP
* Refactor iqk: Factor out float GEMM (AVX2/AVX512)
* Refactor iqk: Factor out GEMM for legacy quants (AVX2/AVX512)
* Refactor iqk: Factor out GEMM for k-quants (AVX2/AVX512)
* Refactor iqk: fix AVX2
* Refactor iqk: Factor out GEMM for i-quants (AVX2/AVX512)
* Refactor iqk: fix AVX2
* Refactor iqk: Factor out GEMM for iqk-quants (AVX2/AVX512)
* Refactor iqk: fix AVX2
* Refactor iqk: Factor out GEMM for 1-bit quants (ABX2/AVX512)
* Refactor iqk: fix AVX2
* Refactor iqk: Factor out GEMM for iq1_bn, iq2_bn, iq2_bn_r4
* Refactor iqk: Factor out GEMM for repacked legacy quants
* Refactor iqk: Factor out GEMM for q8_K_R8, q8_KV
* Refactor iqk: Factor out GEMM for repacked i-quants
* Refactor iqk: GEMM kernels are refactored on AVX2/AVX512
* Refactor iqk: factor out 1-bit quants (NEON)
* Refactor iqk: factor out k-quants (NEON)
* Refactor iqk: factor out floats (NEON)
* Also iq4_xs belongs to k-quants
* Refactor iqk: factor out iqk quants (NEON)
* Refactor iqk: factor out legacy quants (NEON)
* Refactor iqk: factor out repacked legacy quants (NEON)
* Refactor iqk: factor out repacked k-quants (NEON)
* Refactor iqk: factor out repacked iqk quants (NEON)
* Refactor iqk: GEMM kernels are refactored on NEON
* Refactor iqk: FA compiles
If it works is a different story.
Current compile time: 107.3 sesonds on the Ryzen-7950X
* Refactor iqk: FA refactored (Zen4)
Compile time for the FA files is now ~21 seconds on my
Ryzen-7950X, so still slightly too long for my taste
but much better than the 142 seconds we had before.
* Adding forgotten file
* Most helpers don't need to be templates
Also hide Q4_0 and Q8_KV behind IQK_FA_ALL_QUANTS.
Compilation time drops to 14 second on the Ryzen-5975WX
* Fix bf16
* Refactor iqk: FA refactored (NEON)
* Forgotten MMQ ref and typo (#431)
* Adding forgotten iq5_k_r4
* Fix iq4_k_r4 on NEON
* Fix iq4_ks on NEON
It was broken before the refactoring (the shifts were not correctly
applied).
* Fix q8_0 on NEON
* Fix q6_0 K cache
---------
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
Co-authored-by: Nexes the Elder <124105151+Nexesenex@users.noreply.github.com>
Nexesenex
Kawrakow