The memory wall for local LLMs isn’t the model weights — it’s the KV cache, which grows linearly with context length until it dwarfs the weights themselves. The usual escapes are aggressive quantization (quality drops) or context truncation (capability drops).
TurboQuant applies the insight behind vector compression to the KV cache:
compress the keys and values produced at each layer during inference. Two modes
— turbo4 (4-bit) and turbo3 (3-bit) — deliver 3.8–5.1x memory reduction
with negligible quality loss, shipped as pre-built conda binaries (no
compilation).
What that unlocks:
- 128B models on consumer hardware, and 32K+ context where you previously had to truncate to 8K.
- OpenAI-compatible API — the llama.cpp server drops straight into agents and coding assistants.
- Native performance — conda over Docker keeps direct hardware access (Metal on Apple Silicon, CUDA on Linux).
- Multi-platform pre-built packages for macOS, Linux x86_64 (CPU/CUDA) and aarch64.
I run a TurboQuant-accelerated llama.cpp model as the free local tier of my sovereign AI gateway — and cohort participants get to use it directly.
Open source: github.com/kellerkev/turboquant-plus-local.
This is a condensed version. Read the full article on kevinkeller.org →