license: apache-2.0
base_model: huihui-ai/Huihui-Qwen3.8-27B-abliterated
tags:

• gguf
• qwen3.8
• qwen3
• abliterated
• uncensored
• llama.cpp
• quantized
• text-generation
  language:
• en
• ja

Huihui-Qwen3.8-27B-abliterated-Q5_K_M-GGUF

Self-quantized Q5_K_M GGUF of huihui-ai/Huihui-Qwen3.8-27B-abliterated, an abliterated (uncensored) variant of Qwen/Qwen3.8-27B (dense, DeltaNet + gated-attention hybrid architecture).

The official huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF repo currently ships only Q4_K (16.8GB, text + a separate mmproj-model-bf16.gguf vision projector). This repo adds a Q5_K_M quant for users who want higher fidelity and have the VRAM/RAM headroom. This conversion is text-only — no vision/mmproj file is included (we converted directly from the base safetensors without the vision tower).

Vision works by pairing with the official mmproj (verified 2026-08-17): download mmproj-model-bf16.gguf (888MiB, sha256 c9a09064683620bea3d3bfed5d4462e1a97a7d2fff7e5045d6862a0a85eeb5b5) from the official Huihui-Qwen3.8-27B-abliterated-GGUF repo and add --mmproj mmproj-model-bf16.gguf to the launch command. Measured with this Q5_K_M: image understanding works (OpenAI-compatible image_url input), text speed unaffected, n_ctx stays 262,144, one image consumes ~1,500 prompt tokens, VRAM +~1.2GB.

Files

Quantize stats from the conversion log: F16 source 52115.19 MiB (16.00 BPW) → Q5_K_M 18620.21 MiB (5.72 BPW).

How it was made

• Source: huihui-ai/Huihui-Qwen3.8-27B-abliterated, revision fa35f6289942981bffebbc67e94dcb7131ec422e (HF main as of 2026-08-16).
• Converted safetensors → F16 GGUF with convert_hf_to_gguf.py from a ggml-org/llama.cpp checkout at commit 9b05354ec6fb58b4e665e9a39ebc40285c015638 (mainline master, 2026-08-14; this Qwen3.8/"qwen35" architecture support is available on mainline, no fork required).
• Quantized F16 → Q5_K_M with llama-quantize from the same build.
• No imatrix was used for this quant.

Benchmarks (RTX 3090, this repo's own testing)

Direct measurements of this Q5_K_M file, on a machine with 2x RTX 3090 (llama.cpp auto tensor-split across both GPUs), --ctx-size 32768, --flash-attn on -ctk q8_0 -ctv q8_0 --jinja --reasoning off:

• Load time: ~5.7s (server ready, mmap).
• VRAM: ~25.96 GB total across both GPUs (12.97GB + 12.99GB).
• Prompt processing: 122.2 tok/s. Generation: 39.0 tok/s.
• No measurable swap/RAM pressure increase during load or inference.
• Quality spot-check (Japanese, 2 prompts): natural output, no refusals, no degradation vs. the Q4_K_M build of the same base model family.

Context-length ladder (measured on this Q5_K_M file, 2x RTX 3090, 2026-08-17)

Same flags as above (--flash-attn on -ctk q8_0 -ctv q8_0), tensor-split across both GPUs. Generation speed is flat all the way to the model's native maximum:

*measured with a ~32k-token prompt. No swap growth at any rung. n_ctx_train in the GGUF metadata is 262,144, so 262k is the model's true native ceiling — the full native context fits with headroom on a 48GB dual-GPU setup. (Single 24GB GPU: not measured for this Q5_K_M; the 65k rung's 23.1GB total suggests it is near the edge — test before relying on it.)

Reference numbers below are from Q4_K_M of the same architecture (Qwen3.8-27B) on a single RTX 3090 — included for context only, not measured on this Q5_K_M file:

• llama-bench: pp512 = 1380.96–1382.32 tok/s, tg128 = 42.83–42.85 tok/s.
• Server generation held flat (41.6–41.7 tok/s) across ctx 32768 / 65536 / 131072.
• VRAM at ctx=131072 (single 3090): 22.6 GB — leaves the 24GB card's ceiling for Q4_K_M at roughly that context length.
• MTP speculative decoding (--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1, single slot): generation 42.0 → 65.6 tok/s (+56%), 74% draft-token acceptance rate, +0.4GB VRAM. Not yet re-verified against the Q5_K_M file in this repo; expected to behave similarly but unconfirmed.

Because Q5_K_M is ~16% larger than Q4_K_M, a single 24GB card should be expected to fit a somewhat smaller context window than the Q4_K_M figures above. On dual-GPU 48GB setups the full native 262,144 context is verified working (see the ladder table above).

Recommended launch command

Confirmed working line (this is exactly what was used for the benchmark above):

llama-server \
  -m Huihui-Qwen3.8-27B-Q5_K_M.gguf \
  -ngl 99 \
  --ctx-size 32768 \
  --flash-attn on \
  -ctk q8_0 -ctv q8_0 \
  --threads 8 --threads-batch 8 \
  --jinja --reasoning off \
  --port <PORT>

• --reasoning off disables the model's thinking mode (it separates reasoning_content from content when thinking is left on; both work, off was chosen for lower latency in our use case).
• Drop --reasoning off if you want thinking-mode output.
• For higher context, raise --ctx-size up to the native maximum 262144 — validated for this Q5_K_M file on 2x RTX 3090 (32.7GB total VRAM, generation speed unchanged; see the ladder table). On smaller VRAM budgets, confirm fit yourself.

Notes

• Abliterated/uncensored model. This build inherits the reduced refusal behavior of the upstream huihui-ai/Huihui-Qwen3.8-27B-abliterated base — it will comply with a wider range of requests than the stock instruction-tuned model, including ones the stock model would decline. Use accordingly and at your own judgment/responsibility.
• No moderation layer is included or implied. This is a raw base/instruct-style checkpoint, not a safety-tuned deployment.
• Credits: base model and abliteration work by huihui-ai; underlying architecture by Qwen team / Alibaba (Qwen/Qwen3.8-27B). This repo only performs and hosts the Q5_K_M GGUF conversion.
• License follows the upstream model: Apache 2.0.