* common: fix coding mistakes (typos in identifiers, flags and log strings)
Fix misspelled identifiers and user-facing strings across common, server
and model loading:
- allow_ruless -> allow_rules (misspelled identifier used in the allowlist
CLI parsing and the server slot/context code)
- get_formated_timings/get_formated_generation -> get_formatted_*
- 'termionated' -> 'terminated' in the fit-margin assert message
- 'defaulr' -> 'default' in the YAML dump
- 'overriden' -> 'overridden' in tensor buffer type override logs
- 'becausee' -> 'because' in the output-tensor split log
- 'etected NaNs' -> 'detected NaNs' in the imatrix error message
* common: fix comment typos across src, common, include and examples
Fix misspelled words in code comments:
- llama.h: 'typy' -> 'type', 'transfrom' -> 'transform', 'ecoder' ->
'encoder', 'indicies' -> 'indices', 'Intializes' -> 'Initializes'
- common.h: 'embendings' -> 'embeddings', 'pr' -> 'or' in the
fused-indexer-topk comment
- chat.cpp: 'overridde' -> 'override'
- ngram-map: 'occurences' -> 'occurrences', 'stastistics' -> 'statistics'
- speculative.cpp: 'dont'/'inehit' -> 'don't'/'inherit'
- llama-mmap.cpp: 'dont't' -> 'don't'
- llama-model.h: 'hcurrently andle' -> 'currently handle'
- build_gemma3/4.cpp: 'emdeddings' -> 'embeddings'
- examples: 'quantizuation', 'logprobe', 'throught', 'retrun', 'swich',
'convinient', 'temporally' (-> 'temporary'), 'temproal', 'preceed'
* common: remove duplicate definitions and duplicate help entries
- clip-impl.h: drop the second, identical #define TN_FFN_GATE
- common.cpp: remove the duplicate '-t, --threads N' help entry that was
misplaced in the export-lora section (already listed in the general
section)
- common.cpp: merge the two 'embedding' help groups into a single group
so the embedding options are listed together
- llama.cpp: remove the redundant LLAMA_MAX_LAYERS define (llama-hparams.h
already defines the same value and is included by llama.cpp)
* common: fix remaining typos (accomodate, recommanded, occurences, occassionally)
- accomodate -> accommodate in src/llama.cpp comment
- recommanded -> recommended in quantize.cpp user-facing output
- occurences -> occurrences in test-chat.cpp JSON string
- occassionally -> occasionally in vendor/stb/stb_image_resize2.h comment
Note: tokenizer.ggml.seperator_token_id kept as-is to match GGUF spec
* common: remove duplicate help entries
- remove the duplicate '--reasoning-budget N' help entry that was repeated
in the main section (introduced in e0596bf614 'Autoparser - complete
refactoring of parser architecture (PR 1376)')
- remove the second '--parallel-tool-calls' help entry that advertised the
'-ptc' short flag, which belongs to '--print-token-count' (introduced in
e0596bf614 'Autoparser - complete refactoring of parser architecture
(PR 1376)'); the '-ptc' alias was non-functional for '--parallel-tool-calls'
because the parser only binds it to '--print-token-count'
The canonical help entries are kept:
- '--reasoning-budget N' is listed once
- '--parallel-tool-calls' is listed once (without the conflicting '-ptc' alias)
* common: remove duplicate LOG_ENABLE define
- the '#undef LOG_ENABLE / #define LOG_ENABLE() // dummy stub' pair was
repeated verbatim inside the LOG_DISABLE_LOGS section
- remove the second occurrence (introduced in a2588b53e1 'main : log
file (PR 2748)')
* llama-bench: align MLA and attention-max-batch flags with common tools
llama-bench used '--mla-attn' and '--attn-max-batch' while the common
CLI parsing (common/common.cpp) uses '--mla-use' and
'--attention-max-batch' for the same features. This made the flags
inconsistent across tools.
- update the help text to advertise the canonical names
'--mla-use' and '--attention-max-batch'
- keep the old '--mla-attn' and '--attn-max-batch' spellings working
as aliases so existing scripts are not broken
The divergent names were introduced in 3e536b95b0 'Add optional MLA
(PR 188)'.
* fix typos in comments and user-facing strings
- ngram-map.cpp: 'Do we haven a existing' -> 'Do we have an existing'
(introduced in 1cb7e1bf39 'spec : add self speculative decoding,
ngram and refactor (PR 1261)')
- build_mamba.cpp: 'weigth' -> 'weight' (introduced in 8befd92ea5
'Refactor model compute graphs (PR 1651)')
- gguf-split.cpp: 'one of splits have 0 tensors' -> 'one of the splits
has 0 tensors' (introduced in 75b580db0a 'split: allow
--split-max-size option (PR 6343)')
- gguf-split.cpp: 'merged from %d split' -> 'merged from %d splits'
(introduced in 1b5523dc79 'gguf-split: split and merge gguf per
batch of tensors (PR 6135)')
- convert-llama2c-to-ggml.cpp: missing opening quote in the help line,
'(default %s\\')' -> '(default '%s\\')' (introduced in bb9ebb4394
'Adding support for llama2.c models (PR 2559)')
* harmonize British and American spelling to American English
The codebase uses American English (e.g. --embd-normalize, --color),
but a few strings/comments still used British spellings. Unify them:
- 'normalisation' -> 'normalization' in common.h, common.cpp help text
and code comment, and llama-build-context.cpp comment
- 'colorise' -> 'colorize' in the --color help text (common.cpp)
- 'behaviour' -> 'behavior' in a chat.cpp warning and a llama.cpp comment
- also fix 'openai' -> 'OpenAI' capitalization in the embedding help
text and common.h comment (embedding output format is OpenAI-style)
* common: fix help text formatting inconsistencies
- '-smf16'/'--split-mode-f16' and '-smf32'/'--split-mode-f32' help
entries displayed hardcoded 'true'/'false' as the default value;
show the actual state derived from params.reduce_type instead
- '-no-mmad' help entry had 'fused_mmad?' without a space before the
ternary operator
- '--reasoning-tokens' help continuation lines used tab characters for
indentation while the sibling '--reasoning-format' entry uses spaces;
convert to consistent space indentation
* common: revert smf16/smf32 help text default display change
Revert the '-smf16'/'--split-mode-f16' and '-smf32'/'--split-mode-f32'
help entries back to their original hardcoded 'true'/'false' default
display. The change to derive the default from params.reduce_type was
not desired; the split-mode options are legacy and the hardcoded
defaults reflect their intended meaning.
The other formatting fixes in the same area (fused_mmad ternary
spacing and the reasoning-tokens tab-to-space indentation) are kept.
* llama-bench: fix help text column alignment
The --mla-use and --attention-max-batch help lines introduced by the
flag alignment landed one column off from the sibling entries
((default: at column 51 instead of 50). Adjust the padding so all
help lines align.
* common: fix help text defaults for graph-reduce-type and log-format
Mismatch 1: -grt, --graph-reduce-type help shows default "f32", but actual default (common.h:463) is "f16" and llama.cpp uses GGML_TYPE_F16.
Mismatch 2: --log-format help shows default "json", but actual default (common.h:536 log_json=false) is text.
* common: add -ptcall short flag for --parallel-tool-calls
* typo
* Fix gguf-split.cpp splits output naming
With this fix, the initial extension of the source .gguf file is not included in the naming of the output file before the numeration of the splits.
ex:
No more model.gguf-00001-of-00200.gguf
Instead, model-00001-of-00200.gguf
* increase ggml_max_context to 2048
* Revert GGML_MAX_CONTEXTS to 64
* Partial Requant feature for llama-quantize
- Inspired by the recently portcopied --dry-run feature.
- Allows to partially requantize a split quantized .gguf by requantizing only the missing splits in the destination directory.
- Works both for GGUF which are split tensors by tensors, or by group of several tensors (though this one is not very much tested beyond 2 tensors by split).
- Vibe coded.
* Create output directory if it doesn't exist in llama-quantize
* Create output directory if it doesn't exist in gguf-split
* Add exit when directory fails to be created on Windows
* Use std::filesystem
* cleanup
gguf-split : improve --split and --merge logic (#9619)
* make sure params --split and --merge are not specified at same time
* update gguf-split params parse logic
* Update examples/gguf-split/gguf-split.cpp
Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>
Co-authored-by: slaren <slarengh@gmail.com>
---------
gguf-split : add basic checks (#9499)
* gguf-split : do not overwrite existing files when merging
* gguf-split : error when too many arguments are passed
Authored-by: slaren <slarengh@gmail.com>
* common : gpt_params_parse do not print usage
* common : rework usage print (wip)
* common : valign
* common : rework print_usage
* infill : remove cfg support
* common : reorder args
* server : deduplicate parameters
ggml-ci
* common : add missing header
ggml-ci
* common : remote --random-prompt usages
ggml-ci
* examples : migrate to gpt_params
ggml-ci
* batched-bench : migrate to gpt_params
* retrieval : migrate to gpt_params
* common : change defaults for escape and n_ctx
* common : remove chatml and instruct params
ggml-ci
* common : passkey use gpt_params
* Fix --split-max-size
Byte size calculation was done on int and overflowed.
* add tests.sh
* add examples test scripts to ci run
Will autodiscover examples/*/tests.sh scripts and run them.
* move WORK_PATH to a subdirectory
* clean up before and after test
* explicitly define which scripts to run
* add --split-max-size to readme
* llama: llama_split_prefix fix strncpy does not include string termination
common: llama_load_model_from_url:
- fix header name case sensitive
- support downloading additional split in parallel
- hide password in url
* common: EOL EOF
* common: remove redundant LLAMA_CURL_MAX_PATH_LENGTH definition
* common: change max url max length
* common: minor comment
* server: support HF URL options
* llama: llama_model_loader fix log
* common: use a constant for max url length
* common: clean up curl if file cannot be loaded in gguf
* server: tests: add split tests, and HF options params
* common: move llama_download_hide_password_in_url inside llama_download_file as a lambda
* server: tests: enable back Release test on PR
* spacing
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* spacing
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* spacing
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* split: support in llama_model_loader
* avoid copying the entire vector
Co-authored-by: slaren <slarengh@gmail.com>
* split: move llama_tensor_offset to llama_model_loader
* llama_model_loader: PR feedbacks:
- use only one gguf_context for metadata only
- store all ggml_context in a vector as the files and mappings
- store all weights in a vector along with the source tensor
- rename ctx_gguf to meta
- rename ctx_meta to contexts
* avoid copying the entire vector
* Simplify this by making these optional, switch some layer creation tensor optional
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Handle optional tensors
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama_model_loader: fail if backend cannot allocate buffer
* fix mmap buffer management
* llama_model_loader: map file to backend buffer if the allocation succeeds only
* llama_model_loader: only map tensors included in the context
* llama_model_loader: minor, use same variable name for consistency, fix spacing in types cast
* llama_model_loader: fail if any of backend buffer cannot be allocated
* spacing
Co-authored-by: slaren <slarengh@gmail.com>
* fix loop over pointer
Co-authored-by: slaren <slarengh@gmail.com>
* llama_model_loader: if n_tensors declared not equals to loaded tensors in split, throw an exception instead of asserting
* llama_model_loader: ensure mappings vector has the expected size
* llama_model_loader: use at instead of operator[] if this should never add to the map.
* llama_model_loader: immediately add the backend buffer to the model buffers in order to free them if an error occurs in the next allocation. Reserve the expected size.
* llama_model_loader: be sure the model mappings has enough capacity before allocating backend buffer
* llama_model_loader: fix map -> unordered map
* llama_split_prefix: use a clearer version, not pass split path len but dest max len.
Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>
* llama : minor
ggml-ci
* llama : introduce some typedef helpers
* docs: add model shard in hot topic
* llama_model_loader: put mapping in a unique_ptr from the moment it is allocated
Co-authored-by: slaren <slarengh@gmail.com>
* fix llama_split_prefix
---------
Co-authored-by: slaren <slarengh@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>
* gguf-split: split and merge gguf files per tensor
* gguf-split: build with make toolchain
* gguf-split: rename `--split-tensors-size` to `--split-max-tensors`. Set general.split_count KV to all split
* split : minor style + fix compile warnings
* gguf-split: remove --upload not implemented
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Nexesenex
Olivier Chafik
Christian Zhou-Zheng
Georgi Gerganov
Xuan Son Nguyen
Sigbjørn Skjæret
Pierrick Hymbert
DAN™