* Gemma4: WIP
* Gemma4: WIP - runs with totally wrong results
* Gemma4: WIP - add CPU 512, 512 FA
* Gemma4: WIP
It gives a meaningful response in llama-cli, but PPL is still much too
high. Is this due to tokenizer issues?
* Gemma4: this works
I had forgotten the softcap on the final output.
* Remove log
* Gemma4: WIP E4B/E2B
* Gemma4: Q4B/E2B appear to work now
* gemma4: tokenizer fixes
Add HAVE_FANCY_SIMD path that processes 16 rows at a time using 512-bit
operations, combining two R8 groups via _mm512_inserti32x8. Reuses the
existing qx_r8_q8_dot_product 512-bit overload for the inner dot product.
Also updates num_rows for Q8_1 to 16 under HAVE_FANCY_SIMD.
Co-authored-by: Adam Caldwell <accaldwell@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* wip: port MTP architecture
Ports the Multi-Token Prediction (MTP) architecture to the older `llama.cpp` codebase used by `ikllama`.
Changes include:
- Updating `llama_batch` to support `mtp_params`.
- Modifying `llama_decode_internal` (and `encode`) to handle MTP operations (Warmup, Update, Draft).
- Adding public APIs for MTP state management (`llama_set_draft_input_hidden_state`).
- Adapting the embedding extraction logic to skip MTP update passes.
* Refactors `server_slot` to support generic speculative decoding (MTP or Draft Model).
* core: enable hybrid outputs (logits + embeddings) for MTP support
* fix(mtp): correct KV-cache slot finding for updates
* fix(mtp): persist hidden states to prevent context corruption during drafting
* refactor(mtp): clean unused code
* fix(mtp): update server to new functions name
* fix(mtp): fix graph and save hidden state
* mtp: refactor integration, context params and kv cache search
* mtp: fix hidden state extraction and speculative acceptance flow
* server: fix MTP warmup for long prompts and reset token buffer
* llama: refactor MTP operation state to context parameters
* server: fix n_past calculation in MTP acceptance
* llama: fix mtp enable flags
* speculative: refactor MTP to use common_speculative interface
* context: remove unused signatures
* clip: fix deprecated enum-enum conversion warning
* common: fix format string crash in help message
* context: fix mtp activation logic
* llamat: always use the extracted embedding
* llama: get all embeddings to kv cache
* llama: revert logit to not run mtp for not supported arch
* llama: allocate all the n_outputs for MTP
* wip
* server-context: get only the last embedding for hidden state
* ggml-backend: fix array of bounds in debug build
* server-context: run mt kv update to each prompt batch
* revert segmentation fault fixes
* glm-mtp(feat): optimize graph embedding and recursive drafting
* Update nix flake: sync with upstream, fix for newer nixpkgs
- Sync package.nix with upstream: add rocmGpuTargets/useRpc params,
fix env optionals→optionalAttrs, remove unused inputs
- Rewrite devshells.nix to use mkShell directly instead of passthru
- Fix CUDA license check in nixpkgs-instances.nix for list-type licenses
- Update flake inputs (nixpkgs, flake-parts) to latest
- Update references from ggerganov/llama.cpp to ikawrakow/ik_llama.cpp
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Apply nix review suggestions from baileylu121 (#1371)
- Use legacyPackages instead of import to avoid extra nixpkgs instances
- Use inherit (pkgs) stdenv idiom
- Remove unnecessary lib.pipe wrapper in devshells.nix
- Simplify license normalization with lib.toList in nixpkgs-instances.nix
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Update ik_llama-cuda.Containerfile
- DGGML_NATIVE=ON, as most users are building images locally
- Latest llama-swap version
* Update ik_llama-cpu.Containerfile
- DGGML_NATIVE=ON, as most users are building images locally
- Latest llama-swap version
* Update README.md
- DGGML_NATIVE=ON is now default
* Update ik_llama-cuda-swap.config.yaml
- Adapt to the latest version
* Update ik_llama-cpu-swap.config.yaml
- Adapt to the latest version
* Update ik_llama-cuda-swap.config.yaml
- Add model with aliases
* Update ik_llama-cpu-swap.config.yaml
- Add model with aliases
Kawrakow
Adam Caldwell
firecoperana
mcm007
hksdpc255
Samuel Oliveira Alves
Alec Koumjian
Aliez Ren
Nexes the Elder