nicolas
/
hybrid-llama
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
hybrid-llama/turboquant/TURBOQUANT.md

6.7 KiB
Raw Blame History

TurboQuant fork — process & infrastructure

This is the atomic-llama-cpp-turboquant fork of llama.cpp used as the primary inference backend of Atomic Chat. This document describes the branch model, the dev/staging channel, stable releases and the upstream sync procedure. For what the fork changes technically (TurboQuant KV cache, custom quant types, Inkling arch, merge history) see MERGE_NOTES.md.

Branch model

Branch Role Rules
master Stable. What Atomic Chat ships. Changes arrive only via PR from dev. Releases are tagged here.
dev Staging. Feature/fix integration, no strict stability promise. PRs land here first. Every push builds all platforms and republishes the rolling dev-latest prerelease.
upstream Pure mirror of ggml-org/llama.cpp master. Fast-forward only, never contains fork commits. Used as the merge source for upstream syncs.
legacy/master-2025 Archive of the pre-2026 master tip (24cabf4d0). Frozen.
feature/turboquant-kv-cache Former trunk, kept as an alias during the transition. Do not push; will be deleted eventually.

Dev channel (staging binaries)

Workflow: .github/workflows/dev-build.yml.

Every push to dev builds all eleven archives and republishes the dev-latest rolling prerelease with them. PRs into dev — and into master, so the promotion PR reports the same checks — build everything but publish nothing. If some backend fails, dev-latest is still published with the survivors and the notes list what is missing.

Platform Archives
Linux x64 cpu, vulkan, cuda-12.4, cuda-13.3, rocm
Linux arm64 cuda-13.3 — NVIDIA DGX Spark / GB10, sm_121
Windows x64 cpu, vulkan, cuda-12.4, cuda-13.3
macOS arm64 macos-arm64 (Metal)

The CUDA archives bundle their own libcudart/libcublas and are linked with an $ORIGIN RPATH, so they do not need a CUDA toolkit on the target machine — only a recent enough driver. The Linux arm64 archive uses the arm64-SBSA CUDA build; DGX Spark needs driver r580+ for the bundled CUDA 13.3 runtime.

Grab-and-test on any machine:

gh release download dev-latest -R AtomicBot-ai/atomic-llama-cpp-turboquant \
  -p 'llama-turboquant-linux-x64-vulkan.tar.gz'   # or your platform
tar -xzf llama-turboquant-linux-x64-vulkan.tar.gz
./build/bin/llama-server --version    # → version: turboquant-vX.Y.Z (<count>, <sha>)

macOS dev builds are signed but not notarized (release builds are): xattr -dr com.apple.quarantine build/ after unpacking.

Versioning & stable releases

Version format: <upstream-base>-<fork-semver>, e.g. b10018-1.2.0:

  • b10018 — the upstream llama.cpp build the fork is based on (git rev-list --count $(git merge-base master upstream), matching upstream's b#### release tags). Changes only on upstream syncs; the sync PR updates it in TURBOQUANT_VERSION by hand.
  • 1.2.0 — the fork's own semver: major for breaking changes, minor for features (e.g. implementing turbo-ops for a new backend), patch for fixes.

Single source of truth: the TURBOQUANT_VERSION file at the repo root. CMake embeds it via common/build-info.cpp.in; llama-server --version prints version: b10018-1.2.0 (build <count>, commit <sha>). Note that llama_build_info() (the OpenAI API system_fingerprint) intentionally keeps the upstream b<N>-<sha> format — clients parse it.

Cut a release (from an up-to-date, clean master checkout):

# 1. Write the CHANGELOG section for the version you are about to cut,
#    commit it. `verify-version` refuses the release without it.
# 2. Then:
./scripts/turboquant-release.sh patch|minor|major|X.Y.Z

This bumps the fork-semver part, commits release: b10018-X.Y.Z, tags b10018-X.Y.Z and pushes. The tag triggers .github/workflows/release-turboquant.yml: all backends are built (macOS fully notarized) and published as one GitHub release with all archives. verify-version fails the release — in seconds, before three hours of building — if the tag doesn't match TURBOQUANT_VERSION or if CHANGELOG.md has no ## <tag> section.

Release notes

CHANGELOG.md has one section per tag, and the release notes are generated from it verbatim: that section is the release announcement, so write it for whoever downloads the build — what they get, what changed for them, what to watch out for. Everything mechanical (the asset table, the full commit list since the previous tag, versioning boilerplate) the workflow adds by itself, collapsed below the fold. Do not paste a commit dump into the changelog; the notes already carry one.

Consumers: atomic-chat-conf/backends/turboquant-manifest.json entries all point at the same b10018-X.Y.Z tag; asset names (llama-turboquant-<backend>.zip|tar.gz) are unchanged from the legacy scheme, so the Atomic-Chat runtime URL builder needs no changes.

Legacy per-platform releases (turboquant-<platform>-<sha>) are kept for old app versions; do not delete them.

Upstream sync procedure

Small regular syncs instead of 130k-line big bangs:

# 1. Advance the mirror (fast-forward only — zero conflicts by definition)
git fetch upstream            # remote 'upstream' = https://github.com/ggml-org/llama.cpp.git
git push origin upstream/master:refs/heads/upstream

# 2. Merge into a sync branch off dev
git checkout -b sync/upstream-$(date +%Y-%m-%d) origin/dev
git merge origin/upstream     # resolve conflicts HERE, in the sync branch

# 3. PR the sync branch into dev → CI builds every platform
# 4. Test via dev-latest, then PR dev → master as usual

git merge-base origin/master origin/upstream always tells you exactly which upstream commit the fork is based on.

Conflict hot-spots (see MERGE_NOTES.md for history): ggml-cuda.cu/fattn.cu, ggml-vulkan.cpp (SET_ROWS/supports_op), ggml-metal.metal kernel naming, llama-kv-cache.cpp, gguf-py/gguf/constants.py (quant type ids — the fork renumbered Q2_0 to 47; upstream Q2_0 GGUFs are incompatible).

Known constraints

  • Vulkan: turbo3 flash-attn SPIR-V and banded-FA/lightning-indexer kernels are not implemented; those ops are rejected via supports_op (turbo KV cache falls back off on Vulkan). TURBO_WHT / turbo set_rows / GATED_DELTA_NET Vulkan kernels DO exist.
  • Inkling: no MTP/NextN support yet (heads in GGUF are ignored); the fork's MTP subsystem currently serves qwen35/step35/hy-v3. Planned work.
  • Upstream removed -sm row (CUDA multi-GPU split-buffer) — gone since the inkling merge.
  • CUDA-11: no TurboQuant build; the app maps such GPUs to the CPU backend.