Local AI models keep every past message in memory (the "KV cache"), which can eat gigabytes on a long chat. VeloxQuant-MLX shrinks it, so you get longer conversations, or a bigger model, on the Mac you already have.
On a 16 GB MacBook Air, a 64k-token conversation with Llama-3.1-8B needs
8.00 GB normally — with
rabitq it drops to
1.33 GB and fits.
Why VeloxQuant-MLX
01 — Quantizers
Fit bigger models and longer conversations in the RAM you already have — most methods work out of the box, no calibration step required.
02 — Token eviction
Watch your Mac's memory actually go down while you chat — this is the only path that frees RAM you'll see drop in Activity Monitor in real time.
03 — Cross-layer merging
Squeeze out even more headroom on top of everything else — layers share memory instead of duplicating it, and you don't have to change your code to get it.
04 — Metal kernels
Up to 13× faster and up to 98% less peak memory than the plain MLX version — built to run at full speed on your Mac's own GPU, not adapted from something else.
05 — Drop-in API
Try a different method by changing one string, not your whole setup — your model-loading and generation code stays exactly the same.
06 — Multi-model support
Tested on the models you're probably already running — Llama, Mistral, Qwen, Phi, Gemma, Falcon, and vision-language models too — not a narrow demo.
Size it for your Mac
Pick your model, how long a conversation you want, and how much RAM your Mac has. This uses the same calculations as the playground and the command-line sizing tool, so the numbers match what you'd get running it yourself.
Example: Llama-3.1-8B with a 64k-token conversation needs
8.00 GB of memory normally — more than the ~7.5 GB free on a 16 GB Mac. Our RVQ-1bit
setting brings that down to 1.07 GB; VecInfer-1bit to 512 MB. Enable JavaScript to size
your own model, or run the veloxquant-recommend command-line
tool.
Is this you?
Real numbers, real hardware
Every figure below comes straight from the project's own benchmark suite — the same numbers in README.md.
| Metric | Value | Notes |
|---|---|---|
| Max key cache compression | 16× | VecInfer-1bit, head_dim=128 |
| Metal kernel speedup | 13× | quantize_vq at S=2048 (range 6.9–14.7× over S=128–8192) |
| Peak memory reduction | 98% | 729 MB → 12 MB, Falcon3-7B shape |
| RVQ-1bit compression | 7.5× | Near-zero throughput cost |
| SpectralQuant compression | 5.33× | Per-model measured (Qwen2.5-0.5B / Gemma-4-4B), same bit-width |
| RaBitQ full KV compression | 6× | 1-bit keys + MSE-b4 values, Falcon3-7B |
| CommVQ key compression | 64× | RoPE-commutative VQ, D=128, n_cb=4 |
| KIVI-2bit key compression | 5.8× | Per-channel keys / per-token values; measured on Llama-3.2-3B, Qwen2.5-7B, Mistral-7B |
| Production models validated | 12 | Llama, Mistral, Qwen, Phi, Gemma 3/4, Falcon |
| Tests passing | 2347/2347 | Run on every commit |
Compression ratios above are bit-width accounting, not measured resident memory — see the FAQ for what that means for what you'll actually see in Activity Monitor. For attention-correlation accuracy, generation perplexity, and task-level results beyond needle retrieval, see the full benchmarks page →.
Coming soon
A point-and-click app for this tool: pick a model, pick a compression setting, and watch it load and run — no terminal or coding required. It uses the exact same engine as the command-line tool above, so the results are identical, just with a simple interface on top.
You're on the list.
We'll email you as soon as we start building. In the meantime, the CLI works today — see the quickstart above.
Quickstart
mlx_lm code you already haveShown as a diff — the green lines are what you add. Everything else, including how you generate a response, stays exactly the same.
import mlx_lm+ from veloxquant_mlx import KVCacheBuilder, KVCacheConfig model, tokenizer = mlx_lm.load("mlx-community/Llama-3.1-8B-Instruct-4bit") + config = KVCacheConfig(method="turboquant_rvq", bit_width_inlier=1, seed=42)+ caches = KVCacheBuilder.for_model(model, config) response = mlx_lm.generate( model, tokenizer, prompt="Write a 5,000-word analysis of the RLHF literature.", max_tokens=5000,+ prompt_cache=caches, )
These need an extra one-time setup step or apply to a different kind of model. Mostly useful if you're a developer tuning for a specific case.
import mlx_lm from veloxquant_mlx import KVCacheConfig, KVCacheFactory from veloxquant_mlx.allocators.vecinfer import calibrate_smooth_factors, train_codebook model, tokenizer = mlx_lm.load("mlx-community/Qwen2.5-7B-Instruct-4bit") # One-time calibration — run once, cache the results smooth = calibrate_smooth_factors(sample_keys) # [n_heads, head_dim] codebook = train_codebook(sample_keys_flat, n_centroids=256, sub_dim=8) # 16× compression on the key half of the cache — runs faster automatically # when your Mac's GPU supports it config = KVCacheConfig( method="vecinfer", head_dim=128, key_codebook_bits=8, # 256 centroids key_sub_dim=8, # 16× compression at 1 bit/elem smooth_factors=smooth, key_codebook=codebook, use_metal_kernels=None, # None=auto, True=require, False=forbid ) caches = KVCacheFactory.create_for_model(model, config) response = mlx_lm.generate( model, tokenizer, prompt="Write a 5,000-word analysis of the RLHF literature.", max_tokens=5000, prompt_cache=caches, )
from veloxquant_mlx import ( KVCacheBuilder, KVCacheConfig, calibrate_layer_sensitivities, # 1.6s one-time probe allocate_bits_ratequant, # picks a bit budget per layer automatically ) # Step 1 — probe real activations weights = calibrate_layer_sensitivities(model, tokenizer) # Step 2 — decide how much precision each layer gets; average is exact alloc = allocate_bits_ratequant(weights, target_avg_bits=1.5, beta=3.5) # alloc = [1, 2, 1, 1, 3, 1, 2, ...] one number per layer # Step 3 — build per-layer caches config = KVCacheConfig(method="turboquant_rvq", bit_width_inlier=alloc) caches = KVCacheBuilder.for_model(model, config)
from veloxquant_mlx import KVCacheConfig, patch_vlm_kv_cache # Wires the same compressed caches into mlx-vlm (Qwen2-VL, LLaVA, ...) config = KVCacheConfig(method="turboquant_rvq", bit_width_inlier=1, seed=42) patch_vlm_kv_cache(model, config) # Generate as usual — mlx-vlm's API is unchanged.
Vision-language support covers single-prompt generation — integration guide →
FAQ
mlx_lm does.
mlx_lm — same file format, no conversion step. You add the
compression with three lines of code; the model file itself is untouched.
turboquant_rvq — no setup step required, and it's the
default for a reason: it cuts memory use to roughly a third with very little effect on
quality. If you need the smallest possible footprint on certain models (like the
Qwen2.5 or Gemma families), vecinfer reaches about a
sixteenth of the original size, with a one-time setup step. See the
full method list
for the complete decision guide.
Installation
Requires an Apple Silicon Mac (M1 or later) and Python 3.11 or newer.
pip install VeloxQuant-MLX
git clone https://github.com/rajveer43/VeloxQuant-MLX
cd VeloxQuant-MLX
pip install -e ".[dev]"
Prefer an editor integration? Get the VeloxQuant VS Code extension.