library_name: transformers
license: other
license_name: qwen-community-1.0
license_link: LICENSE
pipeline_tag: image-text-to-text
base_model:
- huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated
tags: - abliterated
- uncensored
- exl3
- dgx-spark
- gb10
- qwen3
Huihui-Qwen3.8-Flash-Next-abliterated - EXL3 3.05bpw (integer-pinned recipe)
EXL3 requant of huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated
(also known as Huihui QFN abliterated - the LM layers are abliterated; the 51B n-gram embedding
table, 4B MTP head, and vision tower are byte-identical to the base model).
Built and validated on NVIDIA DGX Spark (GB10, 128GB unified). Serves with the same memory
profile and speed as the stock 3.05bpw EXL3 packs, with the abliterated behavior preserved
through requantization (20/20 on a standard rapid refusal triage).
Why this pack exists
Two facts forced a custom recipe:
- The GB10 EXL3 loader rejects fractional trellis widths. exllamav3 v1.6.x's LP
allocator emits half-integer expert rates (e.g. 3.5bpw) under the mul1 codebook; a
plain-b 3.05conversion produces a pack the loader refuses (trellis dims 56). - The stock 3.05bpw packs (3.0/5.0 mix) are all-integer. This pack pins every tensor
to the stock allocation, extracted from a stock pack's safetensors headers
(bits = last_dim / 16): 75,277 tensors @ 3bpw, 472 @ 5bpw, 2 @ 4bpw - plus
head/vision 5, MTP 3, out_scales always, codebook mul1, calibration 250x2048.
Assembly notes for anyone reproducing: --ngram_file (reusing a pre-quantized n-gram table)
copies the file but does not splice its entries into model.safetensors.index.json -
the 132 ngram_embedding entries and the inclusive total_size must be merged from a pack
that has them. This pack ships fully assembled.
Numbers (DGX Spark, single node, 262k ctx, batch 5)
| Metric | Value |
|---|---|
| Decode, thinking-off prose | ~65 tok/s median |
| MTP2 acceptance | 52-63% (2.26-3.87 tok/round, k=7) |
| Prefill (226k-token needle test) | ~1,560 tok/s |
| Long-context recall | needle recovered at 226,497 tokens |
| Pack size | 84 GB (fits the standard 3.05bpw lane budget) |
| Refusal triage | 20/20 answered, 0 refusals |
| Determinism (temp 0) | identical outputs |
| Vision | verified through the bf16 tower (image QA probe) |
Validation battery also covered tool-call round-trips, streaming, 4-way concurrency, and
chat/reasoning split - all pass. Full methodology in the recipe repo (link below).
Serving
OpenAI-compatible endpoint, EXL3 engine with MTP + n-gram speculative decoding, 262,144-token
context. Engine must run with its working directory at the tree containing src/ptx/ (PTX
kernels resolve relatively).
Deployment shape: single Spark per instance - the full pack and full context fit one GB10
(128GB unified) with no TP. We run two independent single-node serves pooled behind an
OpenAI-compatible router for throughput; each node independently sustains the numbers above.
No multi-GPU choreography required.
Provenance and license
- Base: Qwen/Qwen3.8-Flash-Next - Qwen
Community License 1.0 (carried in LICENSE). - Abliteration: huihui-ai (see source card for their notes and donation links).
- This requant: per-tensor integer recipe pinned to the stock 3.05bpw allocation; n-gram
table, MTP mixer patch, and bf16 vision tower carried through unmodified.
Qwen Community License 1.0 terms apply (included above/at LICENSE, inherited from the source
repo). Usage warnings from the source card apply in full: safety filtering is significantly
reduced; outputs are the user's responsibility.