license: apache-2.0
base_model:
- Qwen/Qwen3.8-27B
tags: - benchmark
- gguf
- mtp
- speculative-decoding
- abliterated
- llama.cpp
Qwen3.8-27B abliterated on an RTX 3080 20 GB: Huihui vs. Heretic RVN, and MTP draft depth
Field measurements, no model weights in this repo. All runs on one RTX 3080 20 GB (Ampere, sm86),
llama.cpp 4df29be4f (b10454) for serving, 46ca246 for llama-perplexity.
Serving flags as in my production setup: --ctx-size 94000 --parallel 1 -ngl 99 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --jinja --mmproj mmproj-model-bf16.gguf --spec-type draft-mtp --spec-draft-ngl 99 --spec-draft-p-min 0. MTP head embedded in all files.
Speed: 400 generated tokens, temperature 0, thinking off, 4 runs per cell (2 prompts × 2), mean oftimings.predicted_per_second; prefill from one 35K-token prompt with cache_prompt: false.
Treat these as indications from one machine, not as a benchmark suite.
Models compared:
| short name | file | size |
|---|---|---|
| Huihui IQ4_XS (my production) | Huihui-Qwen3.8-27B-abliterated-UD-IQ4_XS.gguf |
14.40 GB |
| Heretic RVN IQ4_XS | RVN-IQ4_XS-multilingual-mtp.gguf |
15.53 GB |
| Huihui GSQ IQ3_S | Huihui-Qwen3.8-27B-abliterated-GSQ-RCO-IQ3_S-mtp.gguf |
12.12 GB |
| Heretic RVN GSQ IQ3_S | RVN-Qwen3.8-27B-Heretic-GSQ-RCO-IQ3_S-mtp.gguf |
12.25 GB |
1. MTP draft depth (--spec-draft-n-max), Huihui IQ4_XS
| n-max | code tok/s (acceptance) | prose tok/s (acceptance) | prefill 35K |
|---|---|---|---|
| off | 37.7 | 37.2 | 935 |
| 1 | 56.1 (91 %) | 52.6 (80 %) | 891 |
| 2 | 64.6 (86 %) | 55.1 (65 %) | 891 |
| 3 | 65.9 (77 %) | 51.5 (53 %) | 890 |
| 4 | 67.0 (70 %) | 47.6 (42 %) | 890 |
| 6 | 59.8 (57 %) | 38.3 (31 %) | 889 |
| 8 | 72.9 (48 %) | 44.2 (24 %) | 887 |
- n-max 2 is the best all-round setting here: prose +7 % over n-max 3, code −2 %.
- n-max 8 is the fastest for code but loses 20 % on prose. 6 is worse than both 4 and 8, so
the curve is not monotonic (others have reported a favourable verify shape at 8 as well). - MTP does not help prefill; it costs about 5 %.
2. Perplexity (60 chunks × 512 tokens, lower is better)
German text: Goethe, Die Leiden des jungen Werthers (Project Gutenberg #2407, header/footer
removed). English: wikitext-2-raw test.
| model | German | English |
|---|---|---|
| Huihui IQ4_XS | 10.146 ± 0.225 | 6.317 ± 0.123 |
| Heretic RVN IQ4_XS multilingual | 9.953 ± 0.218 | 6.342 ± 0.123 |
| Huihui GSQ IQ3_S | 10.799 ± 0.241 | 6.457 ± 0.127 |
| Heretic RVN GSQ IQ3_S | 10.723 ± 0.238 | 6.465 ± 0.127 |
Differences between the abliterations are within the error bars. The RVN multilingual file is
about 2 % lower on German, Huihui about 0.4 % lower on English. GSQ IQ3_S costs about 6 % on
German and 2 % on English against IQ4_XS for both.
3. Speed with MTP (n-max 2 and 3)
| model | n-max | code | prose | prefill 35K | VRAM at 94K ctx |
|---|---|---|---|---|---|
| Huihui IQ4_XS | 2 | 64.6 | 55.1 | 891 | 19,435 MiB (n-max 3) |
| Heretic RVN IQ4_XS | 2 | 69.4 | 58.2 | 925 | 19,989 MiB |
| Heretic RVN IQ4_XS | 3 | – | – | – | does not start (OOM) |
| Heretic RVN GSQ IQ3_S | 2 | 64.7 | 53.2 | 872 | 17,371 MiB |
| Heretic RVN GSQ IQ3_S | 3 | 67.0 | 51.0 | 873 | 17,523 MiB |
| Huihui GSQ IQ3_S | 2 | 62.0 | 52.3 | 871 | 17,251 MiB |
| Huihui GSQ IQ3_S | 3 | 67.4 | 50.1 | 870 | 17,401 MiB |
The RVN IQ4_XS run was repeated with identical results (69.6 / 58.3 / 924).
It is the fastest of the four, but on a 20 GB card it leaves only about 470 MiB at 94K context
with the vision projector on the GPU.
4. Full context and tool calls (n-max 2)
One 88,604-token prompt (synthetic numbered paragraphs, question about one of them), then 6 tool-call
prompts (2 tools, single and parallel calls, thinking on and off):
| model | prefill 88.6K | answer | generation at 88.6K | VRAM after | tool calls valid |
|---|---|---|---|---|---|
| Huihui IQ4_XS | 720 tok/s | wrong (70, expected 50) | 14.4 tok/s | 19,313 MiB | 6/6 |
| Heretic RVN IQ4_XS | 742 tok/s | correct (50) | 30.1 tok/s | 20,011 MiB | 6/6 |
Single sample: the retrieval answer and the generation speed at long context come from one prompt
each. Image input on the RVN IQ4_XS at 94K was not tested yet; with 470 MiB headroom that is
the open risk.
Takeaways for this hardware
- Heretic RVN IQ4_XS multilingual is a real alternative to Huihui IQ4_XS: faster with MTP
(+5–7 %), equal or slightly better perplexity on German, held a full 88K context, all tool calls valid.
Cost: 1.1 GB more, which makes 94K + vision on 20 GB very tight and rules out n-max 3. - For more headroom, the GSQ IQ3_S files of both abliterations are about equal and leave ~3 GB free.
- Use n-max 2 rather than 3 unless the workload is mostly code.
Abliterations by Huihui and by 0bserverx (RVN, built on trohrbaugh's heretic-ara); base model by Qwen.