The cache is part of the model's behavior.
Autoregressive generation keeps attention keys and values so each new token can reuse earlier work. That cache grows with the retained context. Hugging Face documents dynamic, static, offloaded, and quantized cache strategies. Its guide also warns that cache quantization can hurt latency when context is short and memory is already sufficient.[2]
The practical choice is not "quantize or do nothing." A serving team can change cache format, move cache data to the CPU, cap context, alter batching, or leave native precision alone. Each move changes a different constraint. The workload decides which constraint matters.
Memory saved is a measurement. Quality retained is another measurement. Throughput is a third.
One format for every layer leaves money on the floor.
NVIDIA's October 8 AutoQuant KV announcement describes a layer-by-layer search. It scores candidate key and value formats against a BF16 reference with KL divergence. An integer program then chooses one candidate per eligible layer under an average-bit budget.[1]
That is the useful idea. Sensitivity is uneven, so the format can be uneven too. The published Qwen3-8B case study kept all value caches in NVFP4. It assigned FP8 keys to 17 of 36 layers at 5.33 effective bits and 26 layers at 5.76 effective bits. The allocation is a result of that model, scoring data, candidate set, and budget. It is not a universal layer recipe.
The headline number has hard edges.
In NVIDIA's case study, the 5.76-bit allocation used 1.62 GiB of modeled KV payload at 32,768 tokens. BF16 used 4.50 GiB. The vendor reports a 64.0 percent reduction in modeled payload and a 1.04 percentage-point drop on RULER-128K. Uniform NVFP4 dropped 6.54 points on that benchmark.[1]
The post states that payload excludes weights and runtime overhead. It does not report total GPU memory or throughput. Results are point estimates, not significance claims. Study checkpoints and the development runtime were not distributed with the post. Those limits belong beside the number.
Calibration and evaluation need different jobs.
The search uses representative prompts to rank format choices. The assembled model still needs held-out tasks. If the same examples choose the layer map and approve it, the test only says the search fit its own evidence.
Record the calibration corpus, score size, sequence lengths, prompt mix, and random seeds. Then freeze the candidate map. Test long-context retrieval, short prompts, code generation, tool calls, and the failure states that cost money in your application. Keep a native-precision baseline in the same serving harness.
Export support is part of the result.
The announcement says the current export path does not support every key and value pairing. It also says a combined weight and KV recipe needs serving support for both sets of formats. An export marked kv_cache_deployment_supported: false still needs runtime work.[1]
vLLM's current documentation covers FP8 cache modes, dataset calibration, per-attention-head scales under the Flash Attention backend, and a way to leave named layers at native precision.[3] Hugging Face supports low-bit cache backends with different limits.[2] A search result and a deployable serving path are separate artifacts.
Road-test the context shape you sell.
Replay the request lengths, batch sizes, concurrency, and output lengths that production sees. Measure cache payload, total device memory, time to first token, decode rate, tail latency, and quality on the same run card. Force an out-of-memory case. Verify the fallback. Restart the server and confirm that the exported scales and layer map reload.
Approve the format only for the tested model, runtime, hardware, and workload envelope. A good result is specific enough to expire when any of those change.