How much RAM does a model need?
Roughly: the size of the GGUF file on disk, plus the KV cache for your chat context, plus about 1.15 GB for Windows and Abliteration itself.
Short answer
Roughly: the size of the GGUF file on disk, plus the KV cache for your chat context, plus about 1.15 GB for Windows and Abliteration itself. A 5 GB model at 8K context needs ~7 GB free RAM to run smoothly.
Detailed explanation
Three things eat RAM when a model runs:
1. Model weights. The GGUF file gets loaded (mostly) into RAM. A 4.5 GB Q4 file needs ~4.5 GB of RAM for the weights themselves. Larger quantizations (Q6, Q8) proportionally larger.
2. KV cache. The model’s short-term memory for the current chat. Scales with context length, layers, and embedding size. For an 8B model at 8K context, ~1 GB. Grows with longer context settings.
3. System overhead. Abliteration uses a fixed budget of ~1.15 GB (400 MB app + 780 MB LLamaSharp runtime), plus whatever Windows and other apps use on top of that.
Formula Abliteration uses on model add: total needed = model file size + KV cache at fixed 8K context + ~1.15 GB overhead. The 8K figure is deliberately fixed - the fit check doesn’t try to predict which context you’ll actually set, because that’s configurable per model in Edit.
Model tier rules of thumb (Q4_K_M quantization, 8K context):
| Tier | File size | KV cache | Overhead | Total RAM |
|---|---|---|---|---|
| 3B | 2.0 GB | 0.5 GB | 1.15 GB | ~3.5 GB |
| 7B | 4.6 GB | 1.0 GB | 1.15 GB | ~7 GB |
| 13B | 9.0 GB | 1.5 GB | 1.15 GB | ~12 GB |
| 30B | 19.5 GB | 2.5 GB | 1.15 GB | ~22 GB |
| 70B | 45 GB | 3.5 GB | 1.15 GB | ~50 GB |
MoE models are an exception - their file is big but only some experts are active per token. See MoE vs dense.
Why it matters
RAM is the #1 bottleneck for local LLM. Get it wrong and you either can’t load the model, or it swaps to disk and generation drops to 0.5 tok/s. Right-sizing means: pick a quant that leaves 2-3 GB of headroom in your free RAM after Windows and other apps.
In Abliteration specifically
The fit-check dialog on model add uses this formula and tells you the number: “Qwen 2.5 7B · Q4 needs about 7 GB. You have 5 GB free right now. PC has 32 GB total.” Then either lets you add anyway (Add anyway) or cancel.
The Hardware card’s Line 3 aggregates this into tier recommendations for your specific PC.
The per-model context slider (⋯ → Edit → context) shows the KV cache change live - moving from 8K to 32K on a 7B model bumps KV from 1 GB to 4 GB.
Related
- Free vs total RAM - why “5 GB free of 32 GB” matters more than just “32 GB”
- KV cache - the growing part of the RAM budget
- Fit-check dialog - the warning that uses this math
- MoE vs dense - MoE breaks the file-size = RAM assumption