nicolas
/
ik_llama_opt
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Do not quantize integer tensors (#2246)

This commit is contained in:
Kawrakow 2026-08-03 10:42:17 +03:00 committed by GitHub
parent 87eeec9f74
commit c013cd876b
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 6 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
src/llama-quantize.cpp
View File

@ -1452,8 +1452,13 @@ static void llama_model_quantize_internal(const std::string & fname_inp, const s
llama_format_tensor_shape(tensor).c_str(),
ggml_type_name(tensor->type));
bool quantize = tensor->type != GGML_TYPE_I32 &&
tensor->type != GGML_TYPE_I64 &&
tensor->type != GGML_TYPE_I16 &&
tensor->type != GGML_TYPE_I8; // i.e., do not quantize tensors holding int values
// This used to be a regex, but <regex> has an extreme cost to compile times.
bool quantize = name.rfind("weight") == name.size() - 6; // ends with 'weight'?
quantize &= name.rfind("weight") == name.size() - 6; // ends with 'weight'?
// quantize only 2D and 3D tensors (experts)
quantize &= (ggml_n_dims(tensor) >= 2);