What Is a KV Cache?

The KV cache is the store of attention keys and values computed for tokens already in the sequence, kept in memory so they’re not recomputed for every new token. It’s what makes autoregressive generation affordable, and it’s a second memory cost sitting alongside the model weights.

Why does the cache exist at all?

Because attention makes each new token look back at every previous one. Without a cache, emitting token 500 would mean recomputing keys and values for the 499 tokens before it. That work produces exactly the same numbers it produced last step, since earlier tokens don’t change.

Why does memory grow as the conversation gets longer?

Because the cache holds one key and one value vector per token, per layer, per attention head. Nothing about it is fixed at load time.

Cost	Scales with	Fixed at load?
Model weights	Parameter count and precision	Yes
KV cache	Sequence length, layers, heads, precision	No
Peak total	Weights plus the longest sequence you allow	No

That second row is the practical consequence: a model that fits comfortably on a device at 500 tokens of history can fail at 8,000. The vLLM authors built PagedAttention around exactly this property, noting that KV cache memory per request is large and grows and shrinks dynamically, which fragments memory and caps batch size.

What does this change for a small model?

It moves the sizing question from “do the weights fit” to “do the weights plus the worst-case cache fit”, the framing used in which SLM fits in 4GB of VRAM.

Two levers help. Shrinking the weights with quantization frees headroom for the cache, and capping how much history you keep bounds the cache directly, which is a decision about your context window.