license: apache-2.0
base_model: abacusai/Smaug-Uncensored
base_model_relation: quantized
pipeline_tag: image-text-to-text
language:
- en
- zh
tags: - gguf
- llama.cpp
- mtp
- agentic
- smaug
- abacusai
1. Introduction
GGUF builds of Smaug-Uncensored for llama.cpp, Ollama and LM Studio: seven sizes from IQ3_M to Q8_0, each as a plain file and as a file that keeps the model's multi-token-prediction (MTP) draft head for faster generation, plus the vision projector for image input. Every file was loaded and tested with thinking on, thinking off and an image, and every plain file plus the MTP Q6_K file was run on GPQA Diamond and IFBench (§4). What the model does and how it was trained is on the main card.
Built with llama.cpp 71a6a11 from the BF16 weights, with an importance matrix from bartowski's calibration_datav3 text (imatrix.gguf, included).
2. Which file to pick
| Quant | Plain file | With MTP head | Mean KLD | 99% KLD | Same top token | Pick it when you have |
|---|---|---|---|---|---|---|
IQ3_M | 11.7 GiB | 11.9 GiB | 0.0611 | 0.628 | 90.3% | Smallest file. 16 GB of GPU memory with a 32K context; a 12 GB card needs part of the model on the CPU. |
Q3_K_M | 12.4 GiB | 12.6 GiB | 0.0521 | 0.527 | 90.7% | 16 GB of GPU memory with a context up to about 24K. Closer to full precision than IQ3_M. |
IQ4_XS | 14.0 GiB | 14.3 GiB | 0.0179 | 0.193 | 94.8% | 16 GB of GPU memory with a short context. |
Q4_K_M | 15.4 GiB | 15.7 GiB | 0.0135 | 0.138 | 95.4% | The default (Ollama picks it). 24 GB of GPU memory. |
Q5_K_M | 17.9 GiB | 18.2 GiB | 0.0055 | 0.054 | 97.1% | 24 GB with room for a longer context. |
Q6_K | 20.6 GiB | 20.9 GiB | 0.0028 | 0.024 | 98.1% | 32 GB, or a 24 GB card with a short context. |
Q8_0 | 26.6 GiB | 27.1 GiB | 0.0009 | 0.009 | 98.8% | Close to full precision. 32 GB and up, or a 48 GB Mac. |
llama-perplexity --kl-divergence; lower is closer. "Same top token" is how often the quant's most likely next token equals BF16's. The MTP files quantize the language model exactly like the plain files (the Q4_K_M pair was checked: all 851 shared tensors are byte-identical) and add the 15 tensors of the draft head. Memory needed is about the file size plus 64 KiB per token of context (16 full-attention layers) plus 0.15 GiB per parallel sequence for the linear-attention state, plus 0.9 GiB for the vision projector when images are used.3. Quick start
llama.cpp (downloads the vision projector too):
llama-server -hf abacusai/Smaug-Uncensored-GGUF:Q4_K_M --jinja -c 32768 \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0
The model thinks by default (reasoning effort xhigh). To turn thinking off, add --chat-template-kwargs '{"enable_thinking": false}', or send "chat_template_kwargs": {"enable_thinking": false} with a request.
Faster generation with the MTP head (llama.cpp): use a file from MTP/ and turn on MTP drafting.
llama-server --hf-repo abacusai/Smaug-Uncensored-GGUF --hf-file MTP/Smaug-Uncensored-27b-MTP-Q4_K_M.gguf \
--mmproj-url https://huggingface.co/abacusai/Smaug-Uncensored-GGUF/resolve/main/mmproj-Smaug-Uncensored-27b-F16.gguf \
--jinja -c 32768 --spec-type draft-mtp --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0
Ollama:
ollama run hf.co/abacusai/Smaug-Uncensored-GGUF:Q4_K_M
LM Studio: search for abacusai/Smaug-Uncensored-GGUF and pick a file from the table above.
4. Checks on the quantized files
GPQA Diamond and IFBench by file
Scored question by question against Q8_0 (paired exact tests), no file differs on GPQA Diamond (p 0.15 or higher). On IFBench every file scores below Q8_0's 86.0, by 1.0 to 5.3 points; the gap is significant for plain Q6_K (80.7, p 0.01) and Q3_K_M (81.0, p 0.04). The MTP Q6_K file carries the same quantized language-model weights as the plain Q6_K and scored 84.0, 3.3 points above it, and the smallest file, IQ3_M, scored 84.7, so part of these single-run gaps is sampling variation.
| Q8_0 plain | Q6_K plain | Q6_K with MTP head | Q5_K_M plain | Q4_K_M plain | IQ4_XS plain | Q3_K_M plain | IQ3_M plain | |
|---|---|---|---|---|---|---|---|---|
| GPQA Diamond 198 questions | 86.4 | 87.4 | 83.8 | 86.9 | 87.4 | 90.4 | 84.3 | 84.8 |
| IFBench 300 prompts | 86.0 | 80.7 | 84.0 | 82.0 | 83.0 | 85.0 | 81.0 | 84.7 |
MTP draft head
The MTP head is inherited from the base model, and it still predicts this model's tokens well. With 3 draft tokens in vLLM 0.32.0, this model accepts 62.2% of drafted tokens (mean accepted length 2.87 tokens per step) against 57.9% (2.74) for the base model, on the same 150 prompts (IFEval, GSM8K, HumanEval) with thinking on. In llama.cpp's smoke test of the MTP Q4_K_M file, 123 of 165 drafted tokens were accepted.
Chat template
llama.cpp renders this repo's embedded chat template exactly as the Hugging Face tokenizer does (checked for a thinking-on turn, a thinking-off turn and a tool-call round trip, on both the plain and the MTP Q4_K_M file), so tool calling and the thinking switch behave as in the BF16 release.
5. License
Apache 2.0, inherited from Qwen3.8-27B. See LICENSE and NOTICE.
6. Citation
@misc{abacusai2026smaugUncensored,
title = {Smaug-Uncensored},
author = {Abacus.AI},
year = {2026},
note = {Fine-tune of Qwen/Qwen3.8-27B with the base model's refusals removed},
url = {https://huggingface.co/abacusai/Smaug-Uncensored}
}
7. Contact Us
If you have any questions, please reach out at Abacus.AI.