d180050f89
* cuda : add the missing compile definitions to the HIP build
GGML_CUDA_FUSION, GGML_CUDA_MIN_BATCH_OFFLOAD and GGML_CUDA_PEER_MAX_BATCH_SIZE
are used unconditionally in common.cuh but were only defined in the CUDA branch,
so every HIP translation unit failed to compile. GGML_CUDA_IQK_FORCE_BF16 and
GGML_CUDA_F16 are user facing options the HIP branch silently ignored.
Also define GGML_USE_HIP. 43 tests in the sources imported from upstream use that
spelling, this fork only defined the older GGML_USE_HIPBLAS, so all of them took
the NVIDIA branch. Without it mmq_id_common.cuh defines TURING_MMA_AVAILABLE,
AMPERE_MMA_AVAILABLE, CP_ASYNC_AVAILABLE and FP16_MMA_AVAILABLE, i.e. the inline
PTX paths, and mmq_id.cu and mmq-instance-q6_k_id.cu then fail to build on
mma_new.cuh:181.
* cuda : shim the warp sync primitives in the HIP vendor header
ROCm 6 and later provide __shfl_sync() and friends as templates that static_assert
on the width of the mask, since an AMD wave can be 64 lanes wide. Define
HIP_DISABLE_WARP_SYNC_BUILTINS so that the shims replace them instead of clashing
with them, and add the shims the header was missing.
This is what makes mmq_id_common.cuh compile, so it unblocks all 26 mmq-instance-*_id
translation units, i.e. the MoE mat-mat path for the iqk quants.
* cuda : update the HIP vendor header for ROCm 6 and 7
- map nv_bfloat16 and nv_bfloat162 onto the __hip_bfloat16 types. Nothing declared
them, so every translation unit that mentions bf16 failed, convert.cu included -
that is the dequantize path the iqk quants use for prompt processing.
- use the hipblasComputeType_t and hipDataType entry points from ROCm 6.5 onwards.
hipblasDatatype_t is deprecated there and no longer matches hipblasGemmEx().
- make cudaStreamWaitEvent object-like. As a 3 argument function-like macro it did
not expand at the 2 argument call sites in reduce.cu, and the unexpanded name was
then passed on to CUDA_CHECK.
- add the mappings for cudaOccupancyMaxActiveBlocksPerMultiprocessor, which 29 of
the 30 failing flash attention translation units needed, and for the entry points
used by dsa_attn.cu and solve_tri.cu. hipBLAS spells a half as an unsigned short,
so cublasHgemmStridedBatched goes through a small casting wrapper rather than a
plain rename, which would not compile at the dsa_attn.cu call site.
* cuda : fix the remaining HIP build errors
- argsort.cu declared the sort order inside #ifdef GGML_CUDA_USE_CUB and used it
outside, and indexer_topk.cu calls argsort_f32_i32_cuda_cub() unconditionally
while it is only defined when CUB is available. Both break any build without CUB,
which includes MUSA and CUDA older than 11.7, not just HIP.
- ggml_backend_cuda_invalidate_graphs() touched ctx->cuda_graphs, which only exists
under USE_CUDA_GRAPH. The function is exported and called from llama-reload.cpp,
so guard the body rather than the function.
- solve_tri.cu included <cublas_v2.h> directly. common.cuh already pulls in whichever
vendor header is right, so the include is removed - it was redundant on CUDA too.
- dsa_attn.cu passed a data type where the GEMM entry point wants a compute type.
hipblasGemmStridedBatchedEx() has no data type taking overload. cuBLAS does, but it
is deprecated: cublas_api.h migrates CUDA_R_32F to exactly CUBLAS_COMPUTE_32F unless
the handle is in CUBLAS_PEDANTIC_MATH, which nothing in this tree sets. So this also
moves the CUDA build onto the primary entry point and drops a cublasGetMathMode()
per call. ggml-cuda.cu already passes a compute type to the same function.
- two mmvq instances called __dp4a() directly instead of ggml_cuda_dp4a(), the
wrapper the rest of the backend uses. On CUDA the wrapper is __dp4a() for every
architecture that has it.
- cap the flash attention vec f32 kernel at 4 columns per block on HIP. Both
logit_softcap variants of the 8 column kernel in one module overflow the 16 bit
branch offset of the AMDGPU backend.
* cuda : build the missing template instances in the HIP build
The HIP source list had drifted from the CUDA one and left out three families of
template instances that the backend references unconditionally:
- mmvq-instance-*.cu, the only definition site for the iqk mat-vec entry points.
iqk_mmvq.cu calls mul_mat_vec_iq4_ks_q8_1_cuda() and mul_mat_vec_iq4_kt_q8_1_cuda()
and nothing defined them, so the library did not link.
- the fattn-vec instances for q8_0-iq4_nl, iq4_nl-iq4_nl, q6_0-q5_0 and q8_0-q6_0,
which fattn-vec-f16.cu and fattn-vec-f32.cu dispatch to in the default
configuration.
- fattn-mma-*.cu. The MMA kernels are never selected on AMD, new_mma_available()
requires an NVIDIA device, but fattn-mma-f16.cu still needs the symbols.
* cuda : recognise AMD GPUs in GGML_CUDA_CC_IS_NVIDIA
CC_OFFSET_AMD is 1000000 and CC_OFFSET_MTHREADS is 0x100000, i.e. 1048576, so the
whole AMD range sits below the Moore Threads offset and every AMD GPU tested as
NVIDIA. turing_mma_available() then returned true on RDNA, the host picked an MMQ
tile of 128 while get_mmq_x_max_device() caps at 64 on AMD, and mul_mat_q_id hit
its NO_DEVICE_CODE guard and wrote NaNs. MUL_MAT_ID on IQ4_KS and IQ4_KT failed
this way on gfx1101.
No effect on CUDA, where a compute capability is 100*major + 10*minor and is
always far below CC_OFFSET_AMD.
* cuda : use v_perm_b32 for the 4 bit table lookup on HIP
HIP implements __byte_perm() in software: it stores an 8 byte union and does four
dynamically indexed byte loads, which end up in scratch. get_int_from_table_16()
calls it eight times per 32 weights, so every quant with a value table was paying
for that, while the trellis types were not.
__builtin_amdgcn_perm() is v_perm_b32, one instruction, and does the same job.
Taken from ggml-org/llama.cpp, which already carries this path.
Token generation on a 7800 XT, pure quantized Qwen2.5-1.5B, tg128:
IQ4_KS 16.30 -> 255.88 t/s
IQ4_XS 17.28 -> 268.05 t/s
IQ4_KT 201.76 -> 195.68 t/s (no table, unchanged)
Perplexity is unchanged to every printed digit and still matches the CPU exactly.
The function is duplicated in vecdotq.cuh and iqk_mmvq_templates.cuh, so both
copies need it - the iqk mat-vec instances only see the latter.
* cuda : use the shared flash attention support check on HIP
supports_op() carried a hand-rolled head size test for HIP that predates the
shared check: it accepted head size 64 with an f16 K cache and head size 128,
and nothing else. Head size 256 was rejected outright, so Gemma-2 and every
other 256 wide model fell back to the CPU for attention even though the
instances are compiled. @hardWorker254 reported 256 working with
ROCm 7.2.4 for both the f16 and the q8_0 cache.
Rather than adding 256 to the list, drop the branch and call
ggml_cuda_fattn_is_supported() as every other backend path does. It already
handles AMD: for cc >= CC_OFFSET_AMD it defers to the vec f16 or vec f32
support predicate depending on precision, which is exactly what
ggml_cuda_flash_attn_ext() dispatches to on AMD, because fast_fp16_available()
is true across the whole AMD cc range. The two now cannot drift.
This also removes a latent abort. The hand-rolled test returned true for any
head size 128 case regardless of the K and V types, so a combination without a
compiled instance, q4_1/q4_1 in a default build, reached the dispatcher and hit
on_no_fattn_vec_case() -> GGML_ABORT instead of falling back to the CPU. The
shared predicate is derived from the instances the build actually contains, and
after the source list repair earlier in this series the HIP build compiles the
same set as the CUDA build.
Beyond head size 256 this also lets HIP claim the asymmetric 192/128 and
576/512 vec f32 paths under GGML_PREC_F32. Those are untested on AMD; they are
gated by the same predicate CUDA uses.
* cuda : test the V head size, not the KV head count, for 192/128 vec f16 FA
ggml_cuda_fattn_vec_f16_is_supported() gates the asymmetric Dk != Dv branch on
if (K->ne[0] != 192 || V->ne[2] != 128) return false;
but ne[2] on K and V is the number of KV heads, not a head size. The test was
meant to be V->ne[0], as the wmma predicate added in the same commit (
|
||
|---|---|---|
| .devops | ||
| .github | ||
| ci | ||
| cmake | ||
| common | ||
| docker | ||
| docs | ||
| examples | ||
| ggml | ||
| gguf-py | ||
| github-data | ||
| grammars | ||
| include | ||
| media | ||
| models | ||
| pocs | ||
| prompts | ||
| requirements | ||
| scripts | ||
| spm-headers | ||
| src | ||
| tests | ||
| vendor | ||
| .dockerignore | ||
| .ecrc | ||
| .editorconfig | ||
| .flake8 | ||
| .gitignore | ||
| .gitmodules | ||
| .mailmap | ||
| .pre-commit-config.yaml | ||
| AUTHORS | ||
| CMakeLists.txt | ||
| CMakePresets.json | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| Package.swift | ||
| README.md | ||
| convert_hf_to_gguf.py | ||
| convert_hf_to_gguf_update.py | ||
| convert_imatrix_gguf_to_dat.py | ||
| convert_llama_ggml_to_gguf.py | ||
| convert_lora_to_gguf.py | ||
| docker-bake.hcl | ||
| docker-bake.override.hcl | ||
| flake.lock | ||
| flake.nix | ||
| llama-mmap.h | ||
| mypy.ini | ||
| poetry.lock | ||
| pyproject.toml | ||
| pyrightconfig.json | ||
| requirements.txt | ||
| test-function-calls.md | ||
README.md
ik_llama.cpp: llama.cpp fork with better CPU performance
TL;DR
This repository started as a fork of llama.cpp in June of 2024 and was last synced with upstream in August of 2024. Compared to mainline llama.cpp, it offers additional SOTA quantization types and, in many cases, better performance. Various features related to LLM inference appeared here first before becoming available in llama.cpp. MLA, quant repacking, fused delta-net (known in `llama.cpp as "Gated Delta Net" - GDN), tensor parallel, MTP, DFlash, to just name a few.
[!IMPORTANT] If you are running hybrid CPU/GPU inference for MoE models with all or some experts left on the CPU, do not use -rtr unless you know what you are doing. The
-rtroption causes all tensors left in RAM to be repacked to row-interleaved format while loading the model. As not all quantization types have a CUDA implementation, this will result in matrix multiplications with these tensors to be always done on the CPU, even when it would have been much better to offload the computation to the GPU, typically resulting in lower prompt processing speed. Most notably, k-quants (K2_K, Q3_K, Q4_K, Q5_K, Q6_K) do not have CUDA row-interleaved implementation.
[!NOTE] The only fully functional and performant compute backends are CPU (
AVX2or better,ARM_NEONor better) and CUDA (Turing or newer). Please do not enter issues related to ROCm, Vulkan, Metal, old Nvidia GPUs,AVXCPUs, etc. They will not get resolved unless you roll up your sleeves and help bring your favorite backend up to speed. With the current regular contributors this project simply does not have the bandwidth to work on all backends available inllama.cpp.
[!IMPORTANT] Do not use quantized models from Unsloth that have
_XLin their name. These are likely to not work withik_llama.cpp.The above has caused some stir, so to clarify: the Unsloth
_XLmodels that are likely to not work are those that containf16tensors (which is never a good idea in the first place). All others are fine.
[!NOTE] Some users have reported issues with graph parallel (a.k.a. split mode
graph) and partial GPU offload (using--cpu-moeor--n-cpu-moeor tensor overrides). If you are using/want to use split mode graph and observe gibberish/incoherent responses, try adding-cuda graphs=0to your command line.
Quickstart
Prerequisites
git clone https://github.com/ikawrakow/ik_llama.cpp
cd ik_llama.cpp
On Debian/Ubuntu Linux, install the required packages (if using another Linux distro, you need to find the corresponding packages and adapt):
apt-get update && apt-get install build-essential git libcurl4-openssl-dev curl libgomp1 cmake
Build for CPU
cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j$(nproc)
For AVX-512-capable CPUs (AMD Zen4 / Intel Sapphire Rapids+), see
docs/build.md section "CPU build flags for AVX-512" for the
additional flags that activate the IQK quantized GEMM kernels (the
HAVE_FANCY_SIMD path). Without those flags, a vanilla Release build
silently falls back to the AVX2 path on this hardware.
Build for GPU
Install Nvidia Drivers and CUDA Toolkit.
cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)
Step-by-step instructions for a case of a successful Windows build
https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md
Run
Download .gguf model files (e.g. bartowski/Qwen_Qwen3-0.6B-IQ4_NL.gguf) to your favorite directory (e.g. /my_local_files/gguf).
Start the server with one of the commands (CPU or GPU):
./build/bin/llama-server --model /my_local_files/gguf/Qwen_Qwen3-0.6B-IQ4_NL.gguf --ctx-size 4096
./build/bin/llama-server --model /my_local_files/gguf/Qwen_Qwen3-0.6B-IQ4_NL.gguf --ctx-size 4096 -ngl 999
That's all! Open http://127.0.0.1:8080 in Browser and start chatting, or use the available API endpoins in your program/harness.
Run in Docker or Podman
Pull one of the available images from ghcr.io. View all tags
docker pull ghcr.io/ikawrakow/ik-llama-cpp:cpu-swap
docker pull ghcr.io/ikawrakow/ik-llama-cpp:cpu-server
docker pull ghcr.io/ikawrakow/ik-llama-cpp:cpu-full
docker pull ghcr.io/ikawrakow/ik-llama-cpp:cu12-swap
docker pull ghcr.io/ikawrakow/ik-llama-cpp:cu12-server
docker pull ghcr.io/ikawrakow/ik-llama-cpp:cu12-full
Check Step by step guide for image customization and other details.
Common parameters and options
Latest News
Model Support
LlaMA-3-Nemotron PR 377, Qwen3 PR 355, GLM-4 PR 344, Command-A PR 341, bitnet-b1.58-2B-4T PR 337, LLaMA-4 PR 321, Gemma3 PR 276, DeepSeek-V3 PR 176, Kimi-2 PR 609, dots.llm1 PR 573, Hunyuan PR 565, GLM-4.5 PR 668 (4.5/4.6/4.7/AIR), Ernie 4.5 MOE and 0.3B PR 759, grok-2 PR 782, Ling/Ring (Bailing-MoE2) PR 833, Qwen3-VL PR 883, SmolLM3 PR 934, GigaChat3 PR 995, ministral3 PR 1030, Mimo-V2-Flash PR 1096, GLM-4.7-Flash PR 1168, Seed-OSS PR 1218, Step-3.5-Flash PR 1231, GLM-5 PR 1268, Qwen3-Next PR 1266, Qwen3.5-MoE PR 1288 and dense Qwen-3.5 1326, Mistral 4 PR 1450, Bonsai 1-bit PR 1570, Gemma4 PR 1581 including assistant, Mimo-2.5 PR 1723, JetBrains Mellum2 PR 1919, Poolside Laguna XS.2 PR 1911, Cohere2-MoE North Mini Code PR 1945, MiniMax-M3 PR 1963, Laguna M.1 PR 2003, OpenPangu #2065, DeepSeek-V4 PR 2165, Muse-Glimmer PR 2293
Quantization
Quantization additions
Trellis quants (IQ1_KT, IQ2_KT, IQ3_KT, IQ4_KT)
Information and the original CUDA implementation in PR 113. Additional implementations: Metal PR 475, Neon PR 471, CPU PR 441. IQ1_KT was added more recently in PR 616. Note: these are base on a novel, integer-base trellis, which allows to achieve reasonable CPU performance, see PR 529 and PRs quoted there for details.
IQK quants
Information can be found in Discussion 8.
Initial implementations (Zen4, AVX2, NEON): IQ5_KS_R4 PR 426, IQ5_KS PR 422, IQ4_KS_R4 PR 150, IQ5_K_R4 PR 149, IQ2_K_R4 PR 146, IQ3_K_R4 PR 145, IQ4_K_R4 PR 138, IQ4_KSS PR 89, IQ2_KS PR 85, IQ4_KS PR 83, IQ6_K PR 14, IQ2_K, IQ3_K and IQ5_K PR 7, IQ4_K PR 6
Cuda implementations: IQ4_KS_R4 and IQ5_KS_R4 PR 493, IQ1_S_R4 PR 492, IQ1_M_R4 PR 494. IQ4_KS_R4 and IQ5_KS_R4 PR 462, IQ2_K_R4, IQ3_K_R4, IQ4_K_R4, IQ5_K_R4 PR 461, IQ4_K, IQ5_K, IQ6_K PR 417, IQ2_KS, IQ2_K, IQ3_K PR 418
IQ2_KL is a more recent addition in PR 602
Hadamard transforms for K-cache
Hadamard transforms for V-cache
MXFP4 as used in gpt-oss models
Implemented for Zen4, AVX2, ARM_NEON, Metal, CUDA PR 682
Quantization improvements
IQ1_MPR 327,IQ2_XSPR 312,Q2_K, Q4_K, Q5_K, Q4_1, Q5_1PR 302,Q4_0, Q5_0, Q6_0, Q3_K, Q6_K, IQ4_XS, IQ4_NLPR 295- Low perplexity
Q4_0KV cache PR 1547 PR 1556 - MTP: option to use re-quantized output tensor
--mtp-requantize-output-tensor new_typePR 1809
Quantization performance improvements
- Much faster CPU prompt processing for all non-interleaved quants. Initial idea in PR 515 and PR 531, with many follow up PRs to apply to all quantization types for the 3 supported CPU platforms.
- All quantization types now have quantized matrix multiplication CUDA kernels, see PR 557 and several others
- Faster CPU prompt processing for Trellis quants and MoE models. PR 488
- Trellis quants: faster CPU prompt processing PR 482.
- Minor (~2%)
iq2_ksTG performance improvement on CUDA PR 468 - Faster
IQ3_KTandIQ4_KTPR 453 - Zen4: Faster PP for
IQ2_KS, IQ4_KS, IQ5_KSPR 428 - Fast GEMM/GEMV for
IQ1_SPR 212 - AVX-VNNI optimizations PR 1446 PR 1455 PR 1467 PR 1474 PR 1482
Features
- New split mode "graph" for multi GPU setups PR 1022
- Fused delta-net for Qwen3-Next and Qwen3.5-MoE PR 1315 PR 1333 PR 1362 PR 1373
- Hadamard transforms for K-cache and V-cache PR 1033 PR 1034 PR 1527
- Auto-fit offloaded tensors to available VRAM (MoE and dense models) PR 1501 PR 1504, allows per GPU fit margin PR 1872
- Checkpoints for recurrent models PR 1310 PR 1398
- MTP decoding support for popular models like GLM-4.x MoE 1270, Qwen 3.5/3.6 1698 1745, Gemma 4 1744, GLM 5 1890, Step 3.7 2250
- Self speculative decoding, ngram PR 1261, suffix PR 1646
- DFlash initial support PR 1970
- DSpark initial support PR 2280
- GLM-DSA architecture indexer cache
- GLM-5.2 vision hack PR 2283
- String ban function for all completions PR 1185 PR 1243
- Expiring Logit Bias PR 1731
- OpenAI
/v1/responsesAPI endpoint PR 1184 - Function call support PR 628
- jinja template support PR 677
- Webui: New Features for Conversations, Settings, and Chat Messages PR 618, MCP PR 1904
- On-demand tensor reload PR 1989
- Dynamic control vector management endpoints PR 1223
- Legacy quants conversion schemes in
convert_hf_to_gguf.pyPR 449,Q6_0in PR 483 - Adaptive-P Sampler PR 1100 implemented as designed by it's author; supported on Webui
- Multi-modal Vision support in
llama-mtmd-cliPR 798 and inllama-serverPR 901 - mikupad as an alternative WebUI PR 558
- June 8 2025: Webui updated (legacy still available when
--path ./examples/server/public_legacyis passed) PR 481 - June 8 2025: RPC improvements PR 480
- June 7 2025: Add an endpoint that lists all the saved prompt caches to server PR 502
- June 6 2025: Make prompt cache saving and restoring MLA aware PR 497
- June 3 2025: Added samplers, XTC PR 486, top-n σ PR 489.
- May 22 2025: Refactor
iqk_mul_mat.cppwhich speeds up compilation time significantly. PR 435 - May 17 2025: Option to enable or disable the CPU FA kernels PR 429.
- May 12 2025: User can now control if/which operations with tensors held in RAM are offloaded to the GPU. See PR 405
- May 12 2025: Compatibility issues with mainline
llama.cppGGUFs for DeepSeek models with MLA enabled were resolved in PR 394. The lower prompt processing performance resulting from usingllama.cpp-style MLA GGUFs was recovered in PR 409. - April 21 2025: ik_llama.cpp builds and runs successfully on Android (using termux), see PR 336
- March 1 2025: Smart Expert Reduction for faster DeepSeek inference PR 239
- Feb 25 2025: Tensor overrides for better control where model weights are stored (GPU or CPU) PR 232
- Feb 23 2025:
sweep-bench- better performance benchmarking PR 225 - Feb 19 2025:
Q8_KV- new type for 8-bit KV-cache quantization PR 208 - March 7 2025: Custom quantization mixes using regular expressions PR 244
Performance improvements
- Better GPU offload strategy for MoE models when using hybrid HPU/CPU inference, see PR 520
- Much faster rng sampling PR 1187
- May 13 2025: Better CPU FA performance for DeepSeek-Lite. PR 410
- May 11 2025: Slightly faster flash attention for DeepSeek models on CUDA, along with extending compatibility to Touring or newer GPUs. PR 408
- May 4 2025: Significant token generation performance improvement on CUDA with Flash Attention for GQA models. For details and benchmarks. PR 370
- April 17 2025: Better CPU Flash Attention token generation performance. PR 332
- April 3 2025: Much faster MoE implementation on Metal. PR 307
- March 25 2025: Better MoE performance on CUDA PR 283
- March 23 2025: Better batched processing speed for DeepSeek models PR 282
- March 18 2025: Reduce compute buffer size PR 237
- March 10 2025: Better TG performance for MoE models on CUDA PR 248
- Feb 23 2025: Fused FFN ops for faster MoE inference PR 229
Flash-MLA
- May 7 2025: 🚀 FlashMLA-3 for DeepSeek models on CUDA. PR 386. Caveat: Ampere or newer Nvidia GPU required
- March 21 2025: 🚀 FlashMLA-3: fastest CPU-only inference for DeepSeek models PR 273
- March 17 2025: 🚀 FlashMLA-2 performance improvements PR 253
- March 12 2025: Allow
Q8_0KV cache with FlashMLA-2 on CUDA PR 265 - March 9 2025: 🚀 FlashMLA on CUDA PR 247
- March 8 2025: 🚀 Faster FlashMLA CPU implementation PR 243
- March 3 2025: 🚀 Introducing FlashMLA - MLA with Flash Attention PR 240
- Feb 27 2025: MLA without transposed cache PR 235
- Feb 13 2025: Allow
Q8_0quantized cache with MLA PR 206 - Feb 11 2025: 🚀 Flash Attention support for DeepSeek models PR 200
- Feb 9 2025: 🚀 MLA for DeepSeek models PR 188
Fixes
- Fix bug in MMVQ kernel PR 446
- Fix AVX2 implementation of
IQ4_K, IQ4_KS, IQ5_K, IQ6_KPR 427 - Fix standard attention on the CPU PR 421
- Fix imatrix calculation for MLA models PR 411
- Fix new CUDA FA on Touring PR 413
- Fix SER. CPU: PR 415 CUDA: PR 416
Resources
There is no single point of reference describing all new ik_llama.cpp features. Pull requests often contain detailed information, so browsing the PRs is often the best way to learn about new features and how to use them. In addition
- The Wiki page has performance comparisons to mainline
llama.cpp - This guide is a good place to start if you came here because of DeepSeek models
- This discussion is about running DeepSeek-V3/R1 on a 16 x 3090 setup
- This discussion describes the new quantization types available in
ik_llama.cpp
Testing
Function Calls Tests
To run the function calls test suite:
cd build
cmake --build . --target test-function-calls
./bin/test-function-calls
The test suite covers parser functionality, streaming, error handling, content cleaning, and server integration. All tests should pass to ensure production readiness.
Contributing
Contributions in form of pull requests, issue submissions (bug reports, feature requests), or general discussions, are welcome.
License
- subprocess.h - Single-header process launching solution for C and C++ - Public domain
- server
- GBNF grammars
Development documentation
Seminal papers and background on the models
If your issue is with model generation quality, then please at least scan the following links and papers to understand the limitations of LLaMA models. This is especially important when choosing an appropriate model size and appreciating both the significant and subtle differences between LLaMA models and ChatGPT:
- LLaMA:
- GPT-3
- GPT-3.5 / InstructGPT / ChatGPT:
Completions
Command-line completion is available for some environments.
Bash Completion
$ build/bin/llama-cli --completion-bash > ~/.llama-completion.bash
$ source ~/.llama-completion.bash
Optionally this can be added to your .bashrc or .bash_profile to load it
automatically. For example:
$ echo "source ~/.llama-completion.bash" >> ~/.bashrc
Dependencies
- yhirose/cpp-httplib - Single-header HTTP server, used by
llama-server- MIT license - stb-image - Single-header image format decoder, used by multimodal subsystem - Public domain
- nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
- miniaudio.h - Single-header audio format decoder, used by multimodal subsystem - Public domain
- subprocess.h - Single-header process launching solution for C and C++ - Public domain