Kawrakow 733660accd Adding ability to have meta data per tensor row (#61) ... * POC: per row scale This is a POC how to work around opinionated ggml to have scales per row rather than per block. Only implemened for Zen4 and only for iq2_tn. * POC per row scale: iq2_tn on NEON * POC per row scale: iq2_tn on Metal * Per row scale Metal templates * iq1_tn: shrink to 1.625 bpw (NEON and Metal) * POC per row scale: CUDA * POC per row scale: add CUDA TODOs There are two places in ggml-cuda.cu left where it is assumed that type_size * n_per_row / block_size is the way to compute and handle row sizes. This does not affect simple usage, but will lead to issues when tensors are split between GPUs. * Per row scales - CUDA The only place left where there are unnecessary assumptions being made is in the Flash Attention code. As we are not using any quants that use per row scales for quantized KV cache, it should be OK for now. * Update IQ1_TN and IQ2_TN bpw shown to user --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>		2024-09-27 08:16:06 +03:00
..
baby-llama	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
batched	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
batched-bench	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
batched.swift	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
benchmark	`build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
convert-llama2c-to-ggml	`build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
cvector-generator	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
deprecation-warning	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
embedding	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
eval-callback	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
export-lora	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
gbnf-validator	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
gguf	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
gguf-hash	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
gguf-split	`build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
gritlm	llama : allow pooled embeddings on any model (#7477)	2024-06-21 08:38:22 +03:00
imatrix	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
infill	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
jeopardy	`build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
llama-bench	Zen4 Flash Attention - bf16 support (#38)	2024-09-05 07:46:47 +03:00
llama.android	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
llama.swiftui	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
llava	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
lookahead	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
lookup	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
main	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
main-cmake-pkg	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
parallel	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
passkey	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
perplexity	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
quantize	Adding ability to have meta data per tensor row (#61)	2024-09-27 08:16:06 +03:00
quantize-stats	quantize_stats: print rmse and max error as fraction of <x> (#21)	2024-08-19 13:49:28 +03:00
retrieval	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
rpc	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
save-load-state	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
server	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
simple	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
speculative	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
sycl	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
tokenize	Merge mainline - Aug 12 2024 (#17)	2024-08-12 15:14:32 +02:00
CMakeLists.txt	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
Miku.sh	`build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
base-translate.sh	`build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
chat-13B.bat	Create chat-13B.bat (#592)	2023-03-29 20:21:09 +03:00
chat-13B.sh	`build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
chat-persistent.sh	`build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
chat-vicuna.sh	`build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
chat.sh	`build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
convert_legacy_llama.py	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
json_schema_pydantic_example.py	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
json_schema_to_grammar.py	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
llama.vim	llama.vim : added api key support (#5090)	2024-01-23 08:51:27 +02:00
llm.vim	llm.vim : stop generation at multiple linebreaks, bind to <F2> (#2879)	2023-08-30 09:50:55 +03:00
pydantic_models_to_grammar.py	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
pydantic_models_to_grammar_examples.py	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
reason-act.sh	`build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
regex_to_grammar.py	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
server-llama2-13B.sh	`build`: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
server_embd.py	Merge mainline llama.cpp (#3)	2024-07-27 07:55:01 +02:00
ts-type-to-grammar.sh	JSON schema conversion: ⚡️ faster repetitions, min/maxLength for strings, cap number length (#6555)	2024-04-12 19:43:38 +01:00