KV Cache, Not Compute, Is What Runs LLM Inference Servers Out of VRAM

Production LLM inference servers usually hit an out-of-memory wall long before they run short on compute, and the culprit is the key-value cache. A new technical writeup from Towards Data Science lays out a VRAM budget formula for serving and maps three optimization strategies to the traffic patterns that trigger the OOM in the first place. The framing matters for anyone sizing GPUs for chat or agent workloads: the bottleneck is memory capacity, not arithmetic throughput.

The KV cache stores the key and value tensors for every token in a conversation so the model does not recompute attention over the full context on each new token. That trade is what makes autoregressive decoding fast, but the cache grows linearly with sequence length and batch size, and it scales with the number of layers, attention heads and head dimension. Model weights are a fixed, predictable cost; the cache is not. At long contexts and high concurrency, it can dwarf everything else sitting in VRAM.

The familiar optimization menu shows up in the analysis: paged attention to cut fragmentation, quantization of the cache to shrink each stored token, and eviction or sliding-window policies to cap how much history stays resident. Each one buys headroom, but each also changes behavior. Quantizing the cache can shift output quality, and evicting tokens changes what the model can still attend to. The writeup's argument is that picking among them requires matching the technique to the traffic pattern, not applying all three by default.

That is where the operational detail gets interesting. A server handling a few very long conversations stresses different limits than one handling thousands of short, bursty requests, even when total token throughput looks similar on a dashboard. Prefill-heavy traffic and decode-heavy traffic also load the cache differently over time. Teams that tune for average utilization can still hit OOM under a burst of concurrent long-context sessions, because the cache allocates per active sequence.

The practical takeaway for serving teams is to budget VRAM explicitly: weights first, then a realistic worst-case cache footprint derived from max context length and expected concurrency, then activation and framework overhead. Metrics like time-to-first-token and inter-token latency hide the memory pressure until it becomes a hard failure. What to watch next is whether frameworks expose per-request cache accounting by default, so operators can see the tax before it takes the server down.

None of this is new physics; attention's memory cost has been known since the transformer paper. What the piece adds is a budgeting discipline that treats the cache as a first-class line item rather than a rounding error. For teams running open-weight models on rented GPUs, that discipline is the difference between a stable endpoint and one that falls over at peak.