llama_token_nl() can return LLAMA_TOKEN_NULL (-1). Falcon3's BPE tokenizer maps
"\n" to zero tokens, so the loader takes its fallback (linefeed_id =
special_pad_id), and the two variants tested reach null by different routes. On
Falcon3-7B-Instruct that copy runs before LLM_KV_TOKENIZER_PAD_ID is read from
the GGUF, so it copies the BPE default, which is itself LLAMA_TOKEN_NULL, even
though the model has a pad token. Falcon3-7B-Base carries no pad id at all and
lands on null whatever the ordering, so a load-order fix alone would not close
this.
llama_sampling_prepare_impl then evaluated logits[-1], an out-of-bounds read one
float before the current position's logit row. Whether that address is mapped
depends on allocation layout, so the crash is configuration-dependent rather
than universal.
This is a crash risk only and cannot change output: the value read is written
back only to a candidate whose id equals nl_token, and no real candidate id is
-1, so it never reaches the sampler.
The fix caches the token once, skips the read when it is null, and skips the
penalize-newline restore, since there is nothing to restore. For a vocab with a
real newline token the block is unchanged.
Repro on Falcon3-7B-Instruct-Q4_K_M, four P100s with the layers split across all
four, -ngl 99 -fa 1 -c 8192, one chat request per trial with a fresh server each
trial: main segfaults 4/4, this change returns HTTP 200 4/4. Across eight models
and both --penalize-nl polarities, 26 greedy comparisons of generated text show
no difference between main and this change on any vocab that has a real newline
token.
Mainline carried this block verbatim until ggml-org/llama.cpp#9294 moved the
penalty stage into the sampler chain, which dropped the raw read and handles the
null id at sampler init instead. ggml-org/llama.cpp#10803 later removed
penalize_nl entirely, so there is no upstream counterpart to port this to.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
8be938842b