nicolas
/
ik_llama_opt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ik_llama_opt/examples
History
Samuel Oliveira Alves 67e6346225
Support for Qwen 3.5 MTP (dense models only) (#1698) 
* qwen-mtp: add dense mtp for one draft

* add support for smaller qwen mtp commit

* qwen-mtp: fix graph for qwen dense variants

* Squashed commit of the following:

commit a92a154b38c7fddc84460f8852c900f8d6ce907e
Author: SamuelOliveirads <samueloliveira32df@gmail.com>
Date:   Mon Apr 20 13:30:21 2026 -0300

    recurrent model: refactor api

commit dfac8f19f6edc0014b4116041b89c1e0dfb173c7
Author: SamuelOliveirads <samueloliveira32df@gmail.com>
Date:   Mon Apr 20 12:22:29 2026 -0300

    recurrent model: implement recurrent kernel checkpoint

commit 9c44b117f93e9060030907e1250106358c9ccf47
Author: SamuelOliveirads <samueloliveira32df@gmail.com>
Date:   Sat Apr 18 11:52:39 2026 -0300

    speculative: fix sampler for checkpoints

commit e7006393bca20adcd86e2d77021e0b41d7bd9db1
Author: SamuelOliveirads <samueloliveira32df@gmail.com>
Date:   Fri Apr 17 14:08:25 2026 -0300

    server: refactor checkpoint state logic

commit 57eabf04df5185cab19539185b8f4b85e578905b
Merge: dc4797b7 64234e3c
Author: SamuelOliveirads <samueloliveira32df@gmail.com>
Date:   Fri Apr 17 13:53:41 2026 -0300

    Merge branch 'main' into fix/hybrid-cache-speculative

commit dc4797b72363482bb35750bb5edc87068116dc0f
Author: SamuelOliveirads <samueloliveira32df@gmail.com>
Date:   Fri Apr 17 13:12:40 2026 -0300

    reset ngram mod state for rejected tokens

commit 8ff2d943a31b3d54440698db41e62ea121d661be
Author: SamuelOliveirads <samueloliveira32df@gmail.com>
Date:   Fri Apr 17 13:08:04 2026 -0300

    server: snapshot recurrent state in tensor

commit d93dfb5e6b78a822e7331b33feedbdc47eb5ec79
Author: SamuelOliveirads <samueloliveira32df@gmail.com>
Date:   Thu Apr 16 22:36:37 2026 -0300

    fix: save/restore sampler state during speculative checkpoint

    When speculative decoding rejects draft tokens and restores the
    recurrent state checkpoint, the sampler (RNG, grammar, prev tokens)
    must also be restored to maintain consistency. Without this, the
    sampler state reflects the rejected draft tokens, leading to
    potential divergence.

    Uses common_sampler_clone() to snapshot the sampler before the
    speculative batch decode, and restores it on rejection.

commit d670cf85cd59f23a339faf6fd773889063176801
Author: SamuelOliveirads <samueloliveira32df@gmail.com>
Date:   Thu Apr 16 21:53:52 2026 -0300

    server: spec checkpoints for recurrent models

* server: fix leak context between requests

* qwen3: allow mtp to run with split graph

* qwen3 mtp: selects rows before the ffn
 		2026-04-28 07:47:50 +02:00
..
baby-llama Merge mainline - Aug 12 2024 (#17) 2024-08-12 15:14:32 +02:00
batched spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
batched-bench spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
batched.swift Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
benchmark Quantization options (#1677) 2026-04-23 09:05:39 +02:00
convert-llama2c-to-ggml spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
cvector-generator spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
deprecation-warning Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
embedding spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
eval-callback spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
export-lora Merge vulkan code from mainline up to commit of 6/28/2025 (#563) 2025-07-02 08:49:42 +02:00
gbnf-validator common : introduce composable PEG parser combinators for chat parsing and new jinja template engine (#1369) 2026-03-09 11:03:33 +01:00
gguf Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
gguf-hash Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
gguf-split gguf-split: fix the split output files naming (#1336) 2026-03-02 08:43:47 +01:00
gritlm spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
imatrix Enable imatrix calculation for models with fused ffn_up/gate_exps tensors (#1418) 2026-03-13 17:57:38 +01:00
infill spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
jeopardy `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
llama-bench Add --defer-experts flag to defer expert mmap residency on Linux (#1634) 2026-04-16 08:54:44 +02:00
llama.android Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
llama.swiftui Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
llava add dry sampler (#513) 2025-06-19 10:24:53 +03:00
lookahead spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
lookup spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
main common : introduce composable PEG parser combinators for chat parsing and new jinja template engine (#1369) 2026-03-09 11:03:33 +01:00
main-cmake-pkg Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
mtmd Vision support for Gemma4 (#1635) 2026-04-16 17:26:31 +02:00
parallel spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
parser Autoparser - complete refactoring of parser architecture (#1376) 2026-04-22 10:04:13 +02:00
passkey spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
perplexity spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
quantize Quantization options (#1677) 2026-04-23 09:05:39 +02:00
quantize-stats Quantization options (#1677) 2026-04-23 09:05:39 +02:00
retrieval spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
rpc Refactor chat and server file (#1062) 2025-12-15 08:27:20 +01:00
save-load-state server: enable checkpoint for recurrent models (#1310) 2026-02-26 06:51:18 +01:00
server Support for Qwen 3.5 MTP (dense models only) (#1698) 2026-04-28 07:47:50 +02:00
simple spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
speculative Autoparser - complete refactoring of parser architecture (#1376) 2026-04-22 10:04:13 +02:00
sweep-bench Cleaner log for adjusted splits (#1494) 2026-03-24 07:49:40 +01:00
sycl Merge mainline - Aug 12 2024 (#17) 2024-08-12 15:14:32 +02:00
tokenize spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
CMakeLists.txt Port mdmd from mainline + Qwen2/2.5-VL support (#798) 2025-09-27 08:45:29 +02:00
Miku.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
base-translate.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
chat-13B.bat
…
chat-13B.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
chat-persistent.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
chat-vicuna.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
chat.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
convert_legacy_llama.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
json_schema_pydantic_example.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
json_schema_to_grammar.py Autoparser - complete refactoring of parser architecture (#1376) 2026-04-22 10:04:13 +02:00
llama.vim llama.vim : added api key support (#5090) 2024-01-23 08:51:27 +02:00
llm.vim llm.vim : stop generation at multiple linebreaks, bind to <F2> (#2879) 2023-08-30 09:50:55 +03:00
pydantic_models_to_grammar.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
pydantic_models_to_grammar_examples.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
reason-act.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
regex_to_grammar.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
server-llama2-13B.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
server_embd.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
ts-type-to-grammar.sh JSON schema conversion: ⚡️ faster repetitions, min/maxLength for strings, cap number length (#6555) 2024-04-12 19:43:38 +01:00