docs: --cache-ram-similarity is a minimum, not a maximum (#2395)
Both the `-crs` help text and docs/parameters.md describe the parameter as a
"max of similarity ... that triggers prompt cache". It is the opposite: a
candidate cache entry is REJECTED when it falls below the value.
server_prompt_cache::load() takes it as `min_reusable_fraction` and applies it as
const float f_keep_cur = float(lcp_cur.first) / tokens.size();
if (f_keep_cur < min_reusable_fraction) {
continue;
}
so raising the value makes reuse stricter, while the current wording reads as
though raising it would admit more. The parameter name in the function signature
already says "min"; only the user-facing strings disagree.
Also states what the fraction is OF, which neither string did: the matched
prefix over the length of the CACHED entry, not over the new prompt.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
3c58ae373a
commit
c2206b80da
|
|
@ -3069,7 +3069,7 @@ void gpt_params_print_usage(int /*argc*/, char ** argv, const gpt_params & param
|
||||||
options.push_back({ "*", "-ctx-ckpt-t N, --ctx-checkpoints-tolerance N", "the number of tokens before the full prompt to create the checkpoint. (default: %d, <=0 disable)",params.ctx_checkpoints_tolerance});
|
options.push_back({ "*", "-ctx-ckpt-t N, --ctx-checkpoints-tolerance N", "the number of tokens before the full prompt to create the checkpoint. (default: %d, <=0 disable)",params.ctx_checkpoints_tolerance});
|
||||||
options.push_back({ "*", "-ctx-ckpt-e NAME, --ctx-checkpoints-eviction NAME", "Eviction strategy for checkpoint. Accepts fifo, variance and auto. Auto defaults to variance. Variance preserves coverage and maintains uniform interval. (default: variance)" });
|
options.push_back({ "*", "-ctx-ckpt-e NAME, --ctx-checkpoints-eviction NAME", "Eviction strategy for checkpoint. Accepts fifo, variance and auto. Auto defaults to variance. Variance preserves coverage and maintains uniform interval. (default: variance)" });
|
||||||
options.push_back({ "*", "-cram, --cache-ram N", "set the maximum cache size in MiB (default: %d, -1 - no limit, 0 - disable)",params.cache_ram_mib });
|
options.push_back({ "*", "-cram, --cache-ram N", "set the maximum cache size in MiB (default: %d, -1 - no limit, 0 - disable)",params.cache_ram_mib });
|
||||||
options.push_back({ "*", "-crs, --cache-ram-similarity N", "max of similarity of prompt tokens to cache tokens that triggers prompt cache (default: %.2f).",params.cache_ram_similarity });
|
options.push_back({ "*", "-crs, --cache-ram-similarity N", "minimum fraction of a cached entry that must match the new prompt for that entry to be reusable (default: %.2f).",params.cache_ram_similarity });
|
||||||
options.push_back({ "*", "-cram-n-min N, --cache-ram-n-min N", "minimum number of the cached tokens that triggers prompt cache (default: %d).", params.cache_ram_n_min });
|
options.push_back({ "*", "-cram-n-min N, --cache-ram-n-min N", "minimum number of the cached tokens that triggers prompt cache (default: %d).", params.cache_ram_n_min });
|
||||||
options.push_back({ "*", "-n, --predict N", "number of tokens to predict (default: %d, -1 = infinity, -2 = until context filled)", params.n_predict });
|
options.push_back({ "*", "-n, --predict N", "number of tokens to predict (default: %d, -1 = infinity, -2 = until context filled)", params.n_predict });
|
||||||
options.push_back({ "*", "-b, --batch-size N", "logical maximum batch size (default: %d)", params.n_batch });
|
options.push_back({ "*", "-b, --batch-size N", "logical maximum batch size (default: %d)", params.n_batch });
|
||||||
|
|
|
||||||
|
|
@ -153,7 +153,7 @@ Note: When the available memory is very limited, turn this option off (`-cram 0`
|
||||||
| Parameter | Description | Default | Notes/Examples |
|
| Parameter | Description | Default | Notes/Examples |
|
||||||
| - | - | - | - |
|
| - | - | - | - |
|
||||||
| `-cram, --cache-ram N` | Set the maximum cache size in MiB | 8192 | -1 = no limit, 0 = disable Very useful when the variations of the same prompt are re-sent to the model (coding agents, etc.). [PR 954](https://github.com/ikawrakow/ik_llama.cpp/pull/954) |
|
| `-cram, --cache-ram N` | Set the maximum cache size in MiB | 8192 | -1 = no limit, 0 = disable Very useful when the variations of the same prompt are re-sent to the model (coding agents, etc.). [PR 954](https://github.com/ikawrakow/ik_llama.cpp/pull/954) |
|
||||||
| `-crs, --cache-ram-similarity N` | Max similarity of prompt tokens to cache tokens that triggers prompt cache | 0.50 | |
|
| `-crs, --cache-ram-similarity N` | Minimum fraction of a cached entry that must match the new prompt for that entry to be reusable. Candidates below this are skipped | 0.50 | |
|
||||||
| `-cram-n-min, --cache-ram-n-min N` | Minimum number of cached tokens that triggers prompt cache | 0 | |
|
| `-cram-n-min, --cache-ram-n-min N` | Minimum number of cached tokens that triggers prompt cache | 0 | |
|
||||||
|
|
||||||
## Sampling
|
## Sampling
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue