KV cache - the model's short-term memory

Short answer

The KV cache is RAM the model uses to remember every token it has already seen in the current chat. The bigger the context, the more RAM the KV cache eats - and it grows every message.

Detailed explanation

Every time a language model generates a token, it needs to “look back” at every previous token in the conversation. To avoid recomputing that history from scratch on every step, the model stores intermediate calculations for each seen token - the keys (K) and values (V) - in a chunk of RAM called the KV cache.

The size of the KV cache scales linearly with three things: the number of tokens in the chat (context length), the number of transformer layers in the model, and the embedding size per layer. As a rough formula: KV cache bytes ≈ 2 × n_layers × n_embd × n_tokens × 2 (the last ×2 is for fp16 precision). For an 8B model at 8K context this is around 1 GB. For a 30B model at 32K context it can be 6-10 GB.

If the model uses GQA (Grouped Query Attention), the cache shrinks by 4-8× because groups of query heads share the same K and V. Abliteration reads this factor from the GGUF header (head_count_kv divided by head_count) so the estimate matches reality instead of assuming the worst case.

The KV cache is separate from the model weights themselves. Loading the model file takes RAM (say, 5 GB for a 7B Q4). Chatting with it adds KV cache RAM on top (say, 1 GB more at 8K context). This is why “the model fits” and “the chat runs comfortably” are two different questions.

Why it matters

If you set the context too high for your free RAM, Abliteration can’t allocate the KV cache and either the model fails to load or Windows starts swapping to disk - which slows generation by 10-100×. Understanding KV cache is the difference between “why is my model suddenly at 0.5 tok/s?” and knowing to lower the context.

In Abliteration specifically

The Model info tooltip (the (i) icon next to a model) shows working memory in tokens (e.g. Working memory: 4096 tokens). The KV cache MB estimate specifically appears in the Edit dialog’s context slider label - as you drag the slider, the label updates live: {ctx} / {max} — uses ~{kv} MB extra RAM. Dragging the slider higher increases KV RAM linearly.

Abliteration’s KV formula is GQA-correct - it reads head_count_kv from the GGUF header and uses that instead of assuming head_count_kv == head_count (which would inflate estimates 4-8× for modern models). This makes the estimate match reality closely on GQA models like Llama 3, Qwen 2.5, and Mistral. The formula does not shrink for MoE - KV cache doesn’t benefit from expert routing.

Related

• GQA - Grouped Query Attention - the reason a 7B model’s KV cache isn’t as huge as it looks
• Context length - the knob that controls KV cache size
• RAM requirements - total RAM = weights + KV cache + overhead