GQA - Grouped Query Attention

Short answer

GQA is a design trick that shrinks the KV cache 4-8× by letting groups of query heads share the same key/value heads. Most modern models (Llama 3, Qwen 2.5, Mistral, DeepSeek) use it.

Detailed explanation

Classic transformer attention gives every “query head” its own dedicated pair of “key head” and “value head”. If a model has 32 attention heads, that means 32 K heads and 32 V heads - and every one of them stores data in the KV cache for every token.

GQA breaks the 1-to-1 rule. Instead of 32 K/V heads, you might have only 8 - and every K/V head is shared by 4 query heads. The model still attends with 32 queries, but the KV cache only stores data for 8 heads instead of 32. That’s a 4× reduction in KV RAM for the same context length.

The compression ratio is head_count / head_count_kv. Llama 3 8B uses 32 query heads and 8 KV heads → 4× reduction. Qwen 2.5 32B uses 40 and 8 → 5× reduction. Some models push it further (Mistral Large, some DeepSeek variants → 8×).

Older models from before 2024 (Llama 2, original Mistral 7B v0.1) do NOT use GQA. Their KV cache is much bigger for the same context, which is why the same 8K context on Llama 2 7B might eat 4 GB of RAM while Llama 3 8B eats 1 GB.

Why it matters

Without knowing GQA, any RAM estimate for the KV cache is worst-case - assuming no compression. If a tool assumes no GQA on a Llama 3 model, it will over-estimate RAM by 4× and refuse to load a model that would actually run fine. This is why “will it fit?” answers vary wildly between tools.

In Abliteration specifically

Abliteration reads head_count and head_count_kv from the GGUF header on model add. The ratio is used directly in the KV cache formula, so estimates match reality. You’ll see the effect if you compare a Llama 2 7B and a Llama 3 8B: same-ish file size, but the Llama 3 needs a fraction of the KV RAM at the same context.

You don’t set GQA anywhere in the UI - it’s a fixed property of the model, baked into the file. Abliteration just uses it transparently.

Related

• KV cache - what GQA compresses
• RAM requirements - GQA is one of the reasons two 7B models can have very different RAM needs