KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks
Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli
arXiv:2606.03458 · 67▲ · cs.LG
View on arXiv →Premium readers get an interactive explainer for this paper — a figure you can poke at, not just read.
What it is
KVarN is a calibration-free KV cache quantizer that combines a Hadamard rotation in the channel dimension with an iterative Sinkhorn-style variance normalization across both the token and channel axes of the K and V matrices before round-to-nearest quantization. It targets the specific error accumulation that happens when the cache is quantized on-the-fly during long autoregressive decoding, rather than the parallel prefill setting most prior methods were tuned for.
Why it matters
Test-time reasoning (long chains of thought) makes the KV cache a memory bottleneck, and existing 2-bit quantizers that look fine on prefill benchmarks quietly degrade accuracy because per-token scale errors compound over thousands of generated tokens. KVarN lets you run a 2.3 bit-per-element cache with close to full-precision accuracy on reasoning and coding tasks, with a claimed 0.18% latency overhead and a vLLM implementation available.
Practical takeaway
If you serve reasoning models with quantized KV caches, watch for accuracy drops that only appear during long generation and not on static retrieval tests. You can try the released vLLM implementation to get near-lossless 2-bit caches, and use the paper's 'pseudo-decode' evaluation (re-quantizing the cache every block and forcing later tokens to read it) to catch error accumulation your current benchmarks miss.
Key result
On AIME24 with Qwen3-4B, KVarN scored 60.0% accuracy at 2.3 bits/element versus 55.5% for KIVI and 56.7% for QuaRot at the same bit budget, close to the 61.1% FP16 baseline. Gains elsewhere are smaller (for example MATH500 79.2% vs 78.9% QuaRot), and results cover three model families (Qwen3-4B, Llama-3.1-8B, Phi-4-14B) averaged over three runs.
Subscribe
Get the next issue.
Free. One email a week. Unsubscribe any time: no account, no dark patterns.