license: apache-2.0
tags:

• qwen
• qwen3.5
• gguf
• quantization
• amd
• vulkan
• rDNA1
• Radeon
• llama.cpp
• local-llm
• abliterated
  base_model:
• Jiunsong/SuperQwen3.8-27b-abliterated
  datasets:
• Jiunsong/SuperQwen3.8-27b-abliterated

SuperQwen3.8-27B (abliterated) — Q3-DOWN-XS GGUF (fully resident on 8 GB RDNA1)

A 7.73 GiB GGUF of the abliterated SuperQwen3.8-27B, quantized with the exact same Q3-DOWN-XS recipe as the base model — proving the recipe generalizes to other weight sets. Fully resident in 8 GB VRAM, zero CPU spill, measured on an AMD Radeon RX 5700 XT (RDNA1, gfx1010, 8 GB).

This is the "recipe generalizes" proof. Same tensor-type map, same imatrix, same flags as guideboardlabs/Qwen3.8-27B-Q3-DOWN-XS-GGUF — applied to a different weight set (abliterated). It runs at the same speed and scores higher on the same capability harness.

The honest comparison: on the same 8-task / 26-point Agon Sprint (medium reasoning, seed 42), the abliterated Q3-DOWN-XS scores 23/26 vs the base Q3-DOWN-XS's 21/26 — a +2 net gain. The gain is concentrated in coding (String Cleaner 4/5 vs base's 0/5); the cost is one pure-reasoning task (Logical Deduction 2/2 → 0/2). So it's a trade, not a strict upgrade: better at code/tool-following, slightly worse at one logical-reasoning task. Speed is identical (24.40 vs 24.48 tok/s, within 0.3%).

Hardware measured on: RX 5700 XT (8 GB, RDNA1) + Ryzen 5 1600, llama.cpp Vulkan build.

Why "Q3-DOWN-XS" (the recipe)

The model is a per-tensor-tuned mix of stock quant types, chosen by testing every combination on the GPU. "DOWN-XS" = the FFN down projections use IQ2_XS while the rest uses a higher-quality mix. This is the same recipe as the base model — the point is that it transfers to other weights unchanged.

How to run it (the recipe)

1. Serve it with stock llama.cpp (Vulkan)

GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1 llama-server \
  -m SuperQwen3.8-27B-abliterated-Q3-DOWN-XS.gguf \
  -c 16384 -np 1 -ngl 999 \
  -ctk q8_0 -ctv q8_0 -fa on \
  --jinja --reasoning off --reasoning-format none \
  --port 8101 --host 0.0.0.0

2. The ONE environment variable that matters

GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM=1

This is the single biggest measured win: +48.6% decode throughput. Without it, part of the hot working set gets allocated in host-visible (PCIe-accessible) VRAM, and the card thrashes through it at ~13 GB/s instead of the resident ~300 GB/s. The fix forces full residency.

3. Set the GPU to COMPUTE DPM profile

# as root
rocm-smi --setperflevel 5        # COMPUTE
# or via sysfs (RX 5700 XT):
echo COMPUTE | sudo tee /sys/class/drm/card0/device/power_dpm_force_performance_level

4. Keep it fully resident (-ngl 999)

Offload every layer. Do NOT let layers spill to CPU — a single CPU layer's latency lands on top of the GPU's, killing throughput.

Why this is reproducible (no custom kernel)

• 100% stock quant types (IQ2_XS, IQ4_XS, IQ2_XXS, IQ1_M, IQ3_S, F32...) — verified, zero custom/RDNA tensor types.
• Stock llama.cpp — the fork is a superset; upstream runs this file unchanged.
• Stock env var + stock AMD setting + stock server flags.

An average user with plain llama.cpp + this file + the env var + COMPUTE DPM gets the same result. That's the point of the recipe.

Build / provenance

• Base: Jiunsong/SuperQwen3.8-27b-abliterated (safetensors, qwen3_5 arch)
• Converted to BF16 GGUF (text-only, --no-mtp), then quantized with llama.cpp llama-quantize using the exact same q3-down-xs.tensor-types.txt map + base imatrix as the base model.
• Capability: Agon Sprint 8-task / 26-point harness, medium reasoning, seed 42, same flags for both models.
• Speed: frozen protocol llama-bench -p 512 -n 128 -r 5, Vulkan0, same build/GPU.

License

Apache-2.0. Model: SuperQwen3.8-27B-abliterated (Apache-2.0). See original for details.