base_model:

• huihui-ai/Huihui-Qwen3.8-27B-abliterated
  base_model_relation: quantized
  license: apache-2.0
  license_link: https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/LICENSE
  library_name: transformers
  pipeline_tag: image-text-to-text
  tags:
• quark
• quantized
• mxfp4
• awq
• qwen3_5
• abliterated
• uncensored
• rocm
• rdna4
• vllm

Huihui-Qwen3.8-27B-abliterated-Quark-AWQ-MXFP4

AMD Quark AWQ + MXFP4 (OCP, group 32, E8M0) quantization
of huihui-ai/Huihui-Qwen3.8-27B-abliterated, produced to run on
AMD RDNA4 (gfx1201) under vllm-radiance.

Vision tower, lm_head and the MTP head are left BF16, so this is still a working
vision-language model — not a text-only quant.

Why this exists

MXFP4 checkpoints that actually land on RDNA4's qualified W4A8/MXFP4 kernels are scarce.
AMD publishes amd/Qwen3.8-27B-Quark-AWQ-MXFP4
for the stock model; this repo is the same recipe applied to the abliterated variant, so
you can have the uncensored weights and the fast kernels. Run on 2x Radeon AI PRO R9700.

Recipe

Replays AMD's exact configuration, read back from their checkpoint's
config.json.orig_with_algo_config, so the result lands on the same kernels:

Layers: 48 decoder layers, hidden 5120. Conversion took 1 h 52 min on one
R9700 (whole model in host RAM, one decoder layer on GPU at a time; peak ~20 GiB VRAM).

The architecture is hybrid: full_attention_interval=4, so only 1 layer in 4 holds a KV
cache and the rest keep a fixed-size recurrent (gated delta net) state. KV is therefore
much cheaper than a dense model of the same size, and --max-num-seqs barely moves VRAM.

Serving

--attention-backend=R4D works on this model: Radiance's R4D attention kernel
requires a GQA ratio of exactly 6, and this checkpoint is 24 q / 4 kv = 6.

Known-good vLLM arguments:

vllm serve <this-repo> \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.90 \
  --max-num-batched-tokens 4096 \
  --kv-cache-dtype fp8 \
  --attention-backend R4D \
  --enable-prefix-caching \
  --mamba-cache-mode align \
  --reasoning-parser qwen3 \
  --trust-remote-code

--mamba-cache-mode align is needed for prefix caching to work with the hybrid layers.

Gotchas

• quantization_config.algo_config is stripped to null in config.json, matching
  AMD's published checkpoint. The pre-strip version is kept as
  config.json.orig_with_algo_config for anyone reproducing the recipe.
• Quark 0.12's preprocess_for_quantization raises on qwen3_5, and the image's Quark
  has no dense qwen3_5 LLMTemplate, so the QConfig is built directly and preprocessing
  is skipped. There is nothing to unfuse in a dense model, so this is safe.
• Cold start under Radiance is ~3 minutes (imports, weight load, torch.compile, HIP
  graph capture). Size your health-check timeouts accordingly; the compile result is
  cached per config, so subsequent starts are much faster.
• vLLM 0.30 returns reasoning in a field named reasoning, not reasoning_content.

Provenance and intent

This is a format conversion only. The abliteration is
huihui-ai's work, not mine, and no additional
guardrail modification was performed — dequantized weights land within a few percent of
the BF16 source. The upstream weights are Apache-2.0 and already public; this repo adds a
quantization that runs on AMD consumer/prosumer hardware.

Because these are abliterated weights they will not refuse much. That is the point of the
upstream model, and it is your responsibility to use them lawfully and to add your own
safeguards if you expose them to other people.

Credits

• huihui-ai — the abliterated source model
• Qwen — the base model
• AMD Quark and amd/Qwen3.8-27B-Quark-AWQ-MXFP4 — the recipe
• magiccodingman/vllm-radiance — the RDNA4 runtime