Adding your first model

Short answer

Download a GGUF file from Hugging Face, drag it into Abliteration (or use Settings → Models → Add Model). If it fits your RAM, it’s ready to use immediately. If not, you’ll see a warning dialog with numbers - cancel or add anyway.

Detailed explanation

Two ways to add a model, same on Windows and macOS:

1. Drag and drop - grab the .gguf file from Explorer (Windows) or Finder (macOS) and drop it onto the Models screen (Settings > Models). Drag-drop is only wired to the Models screen, not to the whole app window.
2. Add Model button (Settings > Models) - clicking it opens a small menu with “Choose file…” which opens the standard file picker for .gguf, and a hint “or drag a .gguf onto this screen”.

What happens on add:

1. Abliteration reads the GGUF header to get architecture, context ceiling, layer count, quantization - all from the file metadata.

2. Estimates RAM needed (file size + KV cache at 8K + ~1.15 GB baseline overhead).

3. Compares to your free RAM. Four possible outcomes:

   • Comfortable fit - silent add, model appears in the list ready to use.
   • “Tight fit” - soft warning when the model fits but leaves ≤2 GB headroom.
   • “This model fits your PC, but you need to free up RAM” - you have enough total RAM but not enough right now; close apps for best performance. Cancel or Add anyway.
   • “This model is bigger than your RAM” - total RAM isn’t enough; may be very slow or fail to load entirely. Cancel or Add anyway.

4. Once added, the model file is copied into Abliteration’s models folder. On Windows that folder is %LOCALAPPDATA%\Caty\models\; on macOS it is ~/Library/Containers/org.abliteration.models/Data/Library/Application Support/Abliteration/models/. The original download can be deleted either way. See Where Abliteration keeps your data for the full layout.

First model recommendations:

If you’re just starting and don’t know what to try:

• Under 8 GB RAM: Llama 3.2 3B Instruct Q4_K_M (~2 GB file) - fast, decent chat quality
• 8-16 GB RAM: Qwen 2.5 7B Instruct Q4_K_M (~4.7 GB file) - knowledgeable, fast on modern CPU
• 16-32 GB RAM: Qwen 2.5 14B Instruct Q4_K_M (~9 GB file) - noticeably smarter
• 32+ GB RAM: Qwen3-30B-A3B (MoE) Q4_K_M (~20 GB) - 30B-class knowledge at ~2B active-parameter speed

See Where to download models for Hugging Face links and how to pick between quant variants.

Why it matters

The fit-check on add is the difference between “download 20 GB, wait 5 minutes for it to fail” and knowing upfront whether it’ll work. If the warning says it won’t fit, believe it - Windows’ pagefile swap can drop you to 0.1 tok/s.

In Abliteration specifically

The fit-check dialog shows the cleaned model name (e.g. Qwen3 30B A3B · Q4), not the raw filename - helpful when you’re not sure which of your downloads matches which model page.

Model name is auto-cleaned on add: Llama-3.2-3B-Instruct-Q4_K_M.gguf becomes Llama 3.2 3B Instruct · Q4. If you don’t like the cleaned name, ⋯ → Edit lets you switch to the original filename or a custom name.

The models folder is Abliteration’s private copy - deleting the original download doesn’t affect the app. To remove a model from Abliteration, use the ⋯ → Delete action in the Models tab, which cleans up the file.

Related

• Where to download models - Hugging Face and quantizers
• Fit-check dialog - the warning explained
• Quantization - picking Q4 vs Q6 vs Q8
• Recommended models by RAM - matched to your rig