Per-model context slider (Auto vs Manual)
Each model has its own context setting.
Short answer
Each model has its own context setting. Auto lets Abliteration pick a safe value based on your free RAM. Manual gives you a slider - drag through power-of-two steps to set the context, with live RAM cost showing next to the label.
Detailed explanation
Every model runs with a context length setting - how many tokens it can hold at once. Bigger context = more memory (KV cache) + slower prefill. Right-sizing this per model matters because different models have different practical needs.
Auto mode (default, “Auto (recommended)”). Abliteration picks a context based on:
- Model’s trained ceiling (
n_ctx_train) - Your current free RAM
- A safety margin
Common Auto values: 4K for tight RAM, 8K for typical laptops, up to 32K on well-resourced machines.
Manual mode. A slider stepping through discrete powers of two:
2048 · 4096 · 8192 · 16384 · 32768 · 65536 · 131072.
The minimum is 2048; the maximum is min(n_ctx_train, 131072) - you can’t go above what the model was trained for, and there’s a hard 128K ceiling for practical RAM reasons.
The label. Shows current setting, cap, and live RAM cost:
{ctx} / {max} — uses ~{kv} MB extra RAM
Example: 8,192 / 131,072 — uses ~640 MB extra RAM
Live RAM warning. If the KV estimate exceeds your free RAM, a red warning appears immediately as you drag. Nothing blocks you from Saving - the warning is advisory.
Where to find it. Models tab → click ⋯ on any model → Edit → context section. Choose Auto (recommended) or Manual with a radio pair.
Per-model, not global. Each model remembers its own context setting in the models folder’s manifest.json. Setting Llama 3B to 8K doesn’t affect Qwen 7B. This matters because a 3B model can afford 32K easily while a 30B model might only afford 4K on the same PC.
When to override Auto. Auto is fine for most cases. Manual matters when:
- Working with long documents - bump specific models up to 32K+
- Tight RAM situation - drop specific models to 4K to leave room for other apps
- Testing model behavior at different context sizes for research
Why it matters
Context is one of the biggest levers on RAM. A 7B model at 4K might need 5.5 GB; the same model at 32K needs 8+ GB. Being able to set it per-model lets you tune each one for its role instead of settling for one compromise value.
In Abliteration specifically
Changes save when you click the Save button (not on close). Next time you load the model, it uses your new context. Chats already active with that model keep their current context until reloaded.
The per-model setting persists across app restarts via manifest.json in the models folder. Reinstalling the app keeps your per-model context choices if you kept the models folder.
Rough page-count equivalents (English text) are visible in the model info tooltip on the model row, not in the Edit slider itself.
Related
- Context length - the concept
- Edit model dialog - where the slider lives
- KV cache - the RAM the slider affects
- n_ctx_train - the ceiling the slider respects