Skip to main content
The KV cache is the running memory of a conversation that a model builds as it processes your prompt and generates a reply — and it’s the reason longer context costs more VRAM even though the model itself doesn’t change size. This page explains what it is and why it grows the way it does; for the mechanics of prefill and decode that build and use it, see How inference works.

​
What the KV cache actually is

“KV” stands for key/value — the internal representations the model computes for every token it has processed so far. Instead of recomputing those representations from scratch for every new token, the model stores them in the KV cache and reuses them, appending a new entry each time a token is added to the sequence. This is what makes generation practical at all. Without a cache, generating token 500 of a reply would mean reprocessing all 499 tokens before it, every single time. With the cache, the model only has to do the new work for each new token, and reuse everything it already computed. The cache is built during prefill (when the model reads your prompt) and extended by one entry per token during decode (when it generates the reply) — see How inference works for how those two phases work.

​
Why longer context costs more memory

The KV cache holds one entry per token, per layer, for as long as that token stays in the active context. That means its size scales directly with:
  • Context length — more tokens in play means more cache entries. A 128K-token conversation holds far more cached state than a 4K-token one.
  • Batch size — if you’re serving multiple requests at once, each one needs its own KV cache, so total cache memory multiplies with concurrent requests.
This is memory on top of the model weights, not instead of them. A model that just barely fits in your VRAM at a short context can run out of memory entirely once you push it to a long conversation or a large document, because the weights stay fixed but the cache keeps growing. See Memory and offloading for how the KV cache fits into the full sizing formula, and the GPU model-fit guide for concrete VRAM tables — this page won’t repeat those numbers.

​
The context-versus-speed tradeoff

Beyond memory, a larger KV cache has a speed cost too. Decode is memory-bandwidth-bound — on every single token generated, the model reads the entire KV cache built so far, in addition to its weights (see How inference works). A bigger cache means more data to read on every step, so:
  • Longer context slows decode down. Generation speed (tokens/sec) tends to drop as a conversation grows, even though the model itself hasn’t changed.
  • Longer context costs more memory, independent of the speed cost — it’s a second, separate bill.
Both effects come from the same root cause: the KV cache scales with context, and everything about decode scales with the KV cache.
There’s no way around this tradeoff by picking a different quantization — quantizing weights and quantizing the KV cache are separate levers, and some backends support KV-cache quantization independently as a way to fit longer contexts. That’s a backend-specific setting; check your runtime’s documentation for support before relying on it.

​
Practical takeaway

When you’re deciding how much context to allocate for a deployment, treat it as a real resource budget, not a free setting. A model advertised with a huge maximum context (128K, 256K) doesn’t mean you should always run it at that limit — size your context to what the task actually needs, and check the real memory cost with the GPU model-fit guide before committing.

​
Go deeper

How inference works

The full prompt lifecycle — prefill, decode, and what goes where in memory.

Memory and offloading

The complete sizing formula, including weights and overhead alongside the KV cache.

​
Once context and memory fit

Sizing your KV cache is part of making a model run well locally. Once it’s running and answering requests on localhost, the next step is reaching it from anywhere else.

Give it a stable endpoint

Tokios doesn’t pick, host, or fine-tune the model — once it runs locally, Tokios turns it into a stable, authenticated endpoint you can call from anywhere.