ik_llama_opt/examples
Samuel Oliveira Alves 1f3e832cb3
Improve mtp acceptance rate (#1499)
* wip: port MTP architecture

Ports the Multi-Token Prediction (MTP) architecture to the older `llama.cpp` codebase used by `ikllama`.

Changes include:
- Updating `llama_batch` to support `mtp_params`.
- Modifying `llama_decode_internal` (and `encode`) to handle MTP operations (Warmup, Update, Draft).
- Adding public APIs for MTP state management (`llama_set_draft_input_hidden_state`).
- Adapting the embedding extraction logic to skip MTP update passes.

* Refactors `server_slot` to support generic speculative decoding (MTP or Draft Model).

* core: enable hybrid outputs (logits + embeddings) for MTP support

* fix(mtp): correct KV-cache slot finding for updates

* fix(mtp): persist hidden states to prevent context corruption during drafting

* refactor(mtp): clean unused code

* fix(mtp): update server to new functions name

* fix(mtp): fix graph and save hidden state

* mtp: refactor integration, context params and kv cache search

* mtp: fix hidden state extraction and speculative acceptance flow

* server: fix MTP warmup for long prompts and reset token buffer

* llama: refactor MTP operation state to context parameters

* server: fix n_past calculation in MTP acceptance

* llama: fix mtp enable flags

* speculative: refactor MTP to use common_speculative interface

* context: remove unused signatures

* clip: fix deprecated enum-enum conversion warning

* common: fix format string crash in help message

* context: fix mtp activation logic

* llamat: always use the extracted embedding

* llama: get all embeddings to kv cache

* llama: revert logit to not run mtp for not supported arch

* llama: allocate all the n_outputs for MTP

* wip

* server-context: get only the last embedding for hidden state

* ggml-backend: fix array of bounds in debug build

* server-context: run mt kv update to each prompt batch

* revert segmentation fault fixes

* glm-mtp(feat): optimize graph embedding and recursive drafting
2026-03-25 10:20:22 +01:00
..
baby-llama Merge mainline - Aug 12 2024 (#17) 2024-08-12 15:14:32 +02:00
batched spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
batched-bench spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
batched.swift Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
benchmark `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
convert-llama2c-to-ggml spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
cvector-generator spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
deprecation-warning Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
embedding spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
eval-callback spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
export-lora Merge vulkan code from mainline up to commit of 6/28/2025 (#563) 2025-07-02 08:49:42 +02:00
gbnf-validator common : introduce composable PEG parser combinators for chat parsing and new jinja template engine (#1369) 2026-03-09 11:03:33 +01:00
gguf Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
gguf-hash Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
gguf-split gguf-split: fix the split output files naming (#1336) 2026-03-02 08:43:47 +01:00
gritlm spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
imatrix Enable imatrix calculation for models with fused ffn_up/gate_exps tensors (#1418) 2026-03-13 17:57:38 +01:00
infill spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
jeopardy `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
llama-bench Graph parallel for Qwen-3.5-MoE (#1347) 2026-03-02 07:48:43 +01:00
llama.android Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
llama.swiftui Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
llava add dry sampler (#513) 2025-06-19 10:24:53 +03:00
lookahead spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
lookup spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
main common : introduce composable PEG parser combinators for chat parsing and new jinja template engine (#1369) 2026-03-09 11:03:33 +01:00
main-cmake-pkg Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
mtmd Fix batch calculation for image processing (#1475) 2026-03-20 10:16:05 +01:00
parallel spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
passkey spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
perplexity spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
quantize Fix misleading quantize error message (#1493) 2026-03-23 13:55:18 +01:00
quantize-stats spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
retrieval spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
rpc Refactor chat and server file (#1062) 2025-12-15 08:27:20 +01:00
save-load-state server: enable checkpoint for recurrent models (#1310) 2026-02-26 06:51:18 +01:00
server Improve mtp acceptance rate (#1499) 2026-03-25 10:20:22 +01:00
simple spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
speculative spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
sweep-bench Cleaner log for adjusted splits (#1494) 2026-03-24 07:49:40 +01:00
sycl Merge mainline - Aug 12 2024 (#17) 2024-08-12 15:14:32 +02:00
tokenize spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
CMakeLists.txt Port mdmd from mainline + Qwen2/2.5-VL support (#798) 2025-09-27 08:45:29 +02:00
Miku.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
base-translate.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
chat-13B.bat Create chat-13B.bat (#592) 2023-03-29 20:21:09 +03:00
chat-13B.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
chat-persistent.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
chat-vicuna.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
chat.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
convert_legacy_llama.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
json_schema_pydantic_example.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
json_schema_to_grammar.py common : introduce composable PEG parser combinators for chat parsing and new jinja template engine (#1369) 2026-03-09 11:03:33 +01:00
llama.vim llama.vim : added api key support (#5090) 2024-01-23 08:51:27 +02:00
llm.vim llm.vim : stop generation at multiple linebreaks, bind to <F2> (#2879) 2023-08-30 09:50:55 +03:00
pydantic_models_to_grammar.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
pydantic_models_to_grammar_examples.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
reason-act.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
regex_to_grammar.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
server-llama2-13B.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
server_embd.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
ts-type-to-grammar.sh JSON schema conversion: ️ faster repetitions, min/maxLength for strings, cap number length (#6555) 2024-04-12 19:43:38 +01:00