Where to download GGUF models

Short answer

Hugging Face is the main source. Search for the model name + “GGUF” (e.g. “Qwen 2.5 7B GGUF”). Trusted quantizers to look for: unsloth, bartowski, lmstudio-community. Download the Q4_K_M file for a good default balance.

Detailed explanation

Hugging Face basics. huggingface.co is the largest hub for open-source models. Model authors and community quantizers upload their weights there. GGUF is the file format Abliteration needs - always look for the “GGUF” suffix or a “-GGUF” repo variant.

Trusted quantizers. For any popular base model, several people convert it to GGUF and re-upload. Not all conversions are equal. Reputable ones:

• unsloth - very active, wide model coverage, includes proper chat templates
• bartowski - high-quality quants with good documentation of trade-offs
• lmstudio-community - official-adjacent, tested with LM Studio (works fine in Abliteration too)
• TheBloke - historically important but now inactive (2023 archives still valid)

If a repo name looks random or has no README, prefer one of the above.

Which file to download. Each GGUF repo has multiple files with different quantizations. As a default:

• Q4_K_M - best balance for most users. Small enough for tight RAM, smart enough for real use.
• Q5_K_M or Q6_K - if you have RAM to spare and want better quality
• Q8_0 - for precision-critical tasks (math, code) if you have plenty of RAM
• Q3_K_M or Q2_K - only if you’re squeezing a bigger model into small RAM and accept quality loss

See Quantization explained for the full picture.

Download tips:

• Files are usually 2-20 GB. On slow connections use a browser that resumes.
• Hugging Face has a “Copy download link” button for direct URLs - you can use wget or curl in a terminal if browser downloads are unreliable.
• Files download as .gguf - don’t rename the extension, Abliteration needs it.

Popular models to start with:

Chat / general use:

• Llama 3.2 (1B, 3B) - Meta’s small models, fast
• Llama 3.1 (8B) - most-tested mid-size
• Qwen 2.5 (3B, 7B, 14B, 32B) - Alibaba, strong across sizes
• Mistral 7B - classic, well-behaved
• Gemma 3 (1B, 4B, 12B, 27B) - Google, recent

Reasoning models:

• DeepSeek R1 distills (various sizes)
• Qwen QwQ 32B - strong reasoning at 32B size

MoE (good RAM/speed ratio):

• Qwen3-30B-A3B - 30B params, ~2B active, very fast
• Mixtral 8x7B - 47B params, ~13B active

Abliterated variants (for Abliteration’s research use case): search for abliterated in the model name - see Finding abliterated models.

Why it matters

Model source matters. Two “Qwen 2.5 7B Q4” GGUFs from different quantizers can behave subtly differently - chat template bugs, wrong stop tokens, missing metadata. Sticking to trusted quantizers minimizes surprises.

In Abliteration specifically

Abliteration reads chat templates from GGUF metadata automatically, so quantizers who include the template correctly (unsloth, bartowski) give you plug-and-play models. Quantizers who skip the template field require the family-based fallback, which usually works but isn’t guaranteed for exotic models.

No download manager built into Abliteration - you download separately in your browser, then drag into the app. This is intentional (keeps the app focused; downloading is a separate concern).

Related

• Adding your first model - what to do with the file after download
• Quantization - picking Q4 vs Q6 vs Q8
• Recommended models by RAM - starting points per rig