nicolas
/
ik_llama_opt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
ik_llama_opt/examples
History
Yap Sok Ann 1c13288164
Fix infinite tool calls loop with `--parallel-tool-calls` (#1679) 
Previously, the end-of-turn token would be added to prompt cache by
`slot.cache_tokens.push_back(slot.sampled)`, without going through
`llama_decode()` first. As such, the token doesn't exist yet in the
actual KV cache. Then, in the next turn, processing of this EOT token
will be skipped since it is already in the prompt cache.

The expected sequence (with Qwen3.5 chat template):

    </tool_call><|im_end|>
    <|im_start|>user
    <tool_response>

The actual sequence that the model sees:

    </tool_call>
    <|im_start|>user
    <tool_response>

As the conversation goes on, the model starts losing the ability to
generate an EOT token after `</tool_call>`, since it hasn't seen this
pattern in the context. With `--parallel-tool-calls`, the grammar
allows `</tool_call>` to be followed by either another tool call or an
EOT token. So, this eventually leads to the model making infinite tool
calls.

We fix this by deferring all the processing of the EOT token to the
next turn.

Fixes #1661
 		2026-04-24 09:48:21 +02:00
..
baby-llama Merge mainline - Aug 12 2024 (#17) 2024-08-12 15:14:32 +02:00
batched spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
batched-bench spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
batched.swift Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
benchmark Quantization options (#1677) 2026-04-23 09:05:39 +02:00
convert-llama2c-to-ggml spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
cvector-generator spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
deprecation-warning Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
embedding spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
eval-callback spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
export-lora Merge vulkan code from mainline up to commit of 6/28/2025 (#563) 2025-07-02 08:49:42 +02:00
gbnf-validator common : introduce composable PEG parser combinators for chat parsing and new jinja template engine (#1369) 2026-03-09 11:03:33 +01:00
gguf Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
gguf-hash Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
gguf-split gguf-split: fix the split output files naming (#1336) 2026-03-02 08:43:47 +01:00
gritlm spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
imatrix Enable imatrix calculation for models with fused ffn_up/gate_exps tensors (#1418) 2026-03-13 17:57:38 +01:00
infill spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
jeopardy `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
llama-bench Add --defer-experts flag to defer expert mmap residency on Linux (#1634) 2026-04-16 08:54:44 +02:00
llama.android Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
llama.swiftui Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
llava add dry sampler (#513) 2025-06-19 10:24:53 +03:00
lookahead spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
lookup spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
main common : introduce composable PEG parser combinators for chat parsing and new jinja template engine (#1369) 2026-03-09 11:03:33 +01:00
main-cmake-pkg Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
mtmd Vision support for Gemma4 (#1635) 2026-04-16 17:26:31 +02:00
parallel spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
parser Autoparser - complete refactoring of parser architecture (#1376) 2026-04-22 10:04:13 +02:00
passkey spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
perplexity spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
quantize Quantization options (#1677) 2026-04-23 09:05:39 +02:00
quantize-stats Quantization options (#1677) 2026-04-23 09:05:39 +02:00
retrieval spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
rpc Refactor chat and server file (#1062) 2025-12-15 08:27:20 +01:00
save-load-state server: enable checkpoint for recurrent models (#1310) 2026-02-26 06:51:18 +01:00
server Fix infinite tool calls loop with `--parallel-tool-calls` (#1679) 2026-04-24 09:48:21 +02:00
simple spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
speculative Autoparser - complete refactoring of parser architecture (#1376) 2026-04-22 10:04:13 +02:00
sweep-bench Cleaner log for adjusted splits (#1494) 2026-03-24 07:49:40 +01:00
sycl Merge mainline - Aug 12 2024 (#17) 2024-08-12 15:14:32 +02:00
tokenize spec : add self speculative decoding, ngram and refactor (#1261) 2026-02-13 19:04:55 +01:00
CMakeLists.txt Port mdmd from mainline + Qwen2/2.5-VL support (#798) 2025-09-27 08:45:29 +02:00
Miku.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
base-translate.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
chat-13B.bat Create chat-13B.bat (#592) 2023-03-29 20:21:09 +03:00
chat-13B.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
chat-persistent.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
chat-vicuna.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
chat.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
convert_legacy_llama.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
json_schema_pydantic_example.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
json_schema_to_grammar.py Autoparser - complete refactoring of parser architecture (#1376) 2026-04-22 10:04:13 +02:00
llama.vim llama.vim : added api key support (#5090) 2024-01-23 08:51:27 +02:00
llm.vim llm.vim : stop generation at multiple linebreaks, bind to <F2> (#2879) 2023-08-30 09:50:55 +03:00
pydantic_models_to_grammar.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
pydantic_models_to_grammar_examples.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
reason-act.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
regex_to_grammar.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
server-llama2-13B.sh `build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809) 2024-06-13 00:41:52 +01:00
server_embd.py Merge mainline llama.cpp (#3) 2024-07-27 07:55:01 +02:00
ts-type-to-grammar.sh JSON schema conversion: ⚡️ faster repetitions, min/maxLength for strings, cap number length (#6555) 2024-04-12 19:43:38 +01:00