license: other
license_name: qwen-community-1.0
license_link: LICENSE
base_model:

• huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated
  base_model_relation: quantized
  pipeline_tag: text-generation
  language:
• en
• zh
  tags:
• nvfp4
• gptq
• strata
• moe
• qwen3.8
• flash-next
• abliterated
• uncensored
• mtp

Huihui-Qwen3.8-Flash-Next-abliterated — NVFP4 (GPTQ) for Strata

huihui-ai's abliterated Qwen3.8-Flash-Next
with every routed expert in NVFP4, quantized from the BF16 checkpoint by GPTQ (not by rounding each weight to the
nearest value), packed for Strata NVFP4: the 125B hybrid MoE on one RTX
20-50 card (12 GB of VRAM or more) and 64 GB of RAM or more.

Quick start (Windows): from the Strata NVFP4 release, run
START-HERE.bat --family huihui-nvfp4: it checks the PC, downloads this repository and starts the model. That works
once the fork's installer lists this repository (its next release); until then, see Run it.

• Chain: Qwen/Qwen3.8-Flash-Next →
  huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated
  (abliteration, BF16) → this quantization.
• 4.5 bits a weight for the experts, 63.3 GiB: the size and the decode cost of a ModelOpt NVFP4 checkpoint, about
  34% of plain rounding's error on held-out tokens (below). No layer is kept in 8 bits.
• Only for Strata NVFP4. The files are Strata's own runtime formats (an expert pack, GGUFs the fork reads, the MTP
  draft head). They do not load in transformers, vLLM or llama.cpp.

The model does not refuse. It is an abliterated model: the refusal direction was removed from the weights
(see the base model's card). It will
follow harmful requests. What it is used for is on whoever runs it.

Files

What huihui's abliteration changed, compared tensor by tensor with the BF16 checkpoints: the weights that write into
the residual stream, in every layer - attention o_proj (12 layers), DeltaNet out_proj (36), the shared expert's
down_proj and every routed expert's down_proj (48). Which parts depend on it:

• Model-specific: the experts (down_proj edited in every layer; gate_up_proj is Qwen's, but GPTQ fits the
  codes to this model's activations), the dense GGUF and pack/dense.bin (huihui's o_proj, out_proj and shared
  down_proj, Qwen's everything else).
• The same as Qwen's original: the token embedding and the MTP draft head (every tensor byte-identical to
  Qwen/Qwen3.8-Flash-Next's, compared in full), the n-gram table (huihui's and OrcaRouter's are byte-identical, and
  ple-fp8.gguf is the very file the OrcaRouter NVFP4 repository ships: the same SHA-256), the experts'
  gate_up_proj, the routers and lm_head (compared on samples), the tokenizer, the chat template, config.json
  and the license file. These are here so the repository is complete on its own.

The dense GGUF is llama.cpp's converter run on the BF16 checkpoint with the fork's tools/nvfp4_convert.py type
policy (the projections Q8_0, the small ones BF16, the PLE convolution F16, the n-gram table and the MTP head left to
their own files), with the routed experts left out: next to pack/experts.bin the engine never reads them. Its
tensors are those of the OrcaRouter repository's orca-nvfp4-dense.gguf less the experts' per-matrix scales (the
pack carries its own), of the same types and shapes, and where neither abliteration edited a tensor the bytes are
the same.

How it was quantized

NVFP4 stores a weight as a 4-bit E2M1 code times its 16-value block's FP8 (E4M3) scale times one FP32 scale per
expert matrix. ModelOpt's NVFP4 scales each block to its largest value and rounds every weight to the nearest code
(RTN). Here each expert was quantized from the BF16 checkpoint with tools/requant.py --method gptq from the fork:

• Calibration: the engine's own MoE inputs at every layer (STRATA_DUMP_MOE_INPUT, STRATA_DUMP_MOE_LAYER=all)
  for 57.5K tokens: Ukrainian and English Wikipedia, llama.cpp and Rust sources, and 16 chat answers
  (data/requant_calib in the fork), with the routing of each token, from a run of this model with its experts
  rounded to NVFP4 by ModelOpt's recipe (requant.py pack --method rtn).
• Hessians: for each expert, H = sum w^2 x x^T over the tokens routed to it (w its routing weight: its
  output enters the residual times w), mixed with the layer's average input as 256 pseudo-tokens so rarely used
  experts get a sensible one. down's Hessian is built from silu(gate x) * up x of the already quantized
  gate/up, so down corrects the error gate/up left.
• GPTQ: columns left to right in blocks of 128, 1% damping; each column's rounding error is spread over the
  columns not yet quantized through the inverse Hessian.
• Block scales: each 16-column group's FP8 scale is chosen on the weights as GPTQ has updated them, among the
  codes for amax -> 6 and amax -> 4 ("four over six") and two neighbours of each, by the least squared error
  weighted by diag(H). The FP32 scale per expert matrix leaves 25% headroom so the updates do not saturate E4M3.
• Every routed expert is NVFP4 (no 8-bit layers): the pack is as large as a ModelOpt one and a decode step costs
  the same. The shared experts, attention and DeltaNet projections are Q8_0 in the GGUF.

Error against BF16

requant.py analyze: per layer, on held-out tokens (every 4th calibration token, not used for the Hessians), the
routed-weighted squared error of the experts' whole output down(silu(gate x) * up x) against BF16, relative to the
output's energy; summed over the 48 layers weighted by each layer's energy:

48 layers, 57551 tokens per layer (14387 held out)

GPTQ below RTN in 48 of 48 layers; ratio GPTQ/RTN per layer: min 0.20, median 0.39, max 0.46

Checks

Strata NVFP4 0.1.40.3-nvfp4.1 on an RTX 5090 (32 GB) with 128 GB of RAM, the arguments under Run it
(--spec 6 --spec-min-p 0.7).

• It answers. "Explain in detail how a hash table works ... at least 300 words" (no thinking): a correct
  476-token answer that ends by itself (EOS), 182.8 tok/s. A 2,100-token prompt (a CUDA source file, cut off
  mid-line): prefill 2,810 tok/s; the model finishes the line (... q2[l0 / 2] >> 9);) and closes the turn, and
  in a 200-token greedy run it goes on to describe, in its thinking, what the file implements.

• A decode round costs the same as with plain rounding (RTN) of the same experts. Three interleaved pairs on
  the hash-table question, each answer to its end (--stop-eos), expert cache auto (8,343 slots in every run),
  only experts.bin different:

  experts	ms per round	tokens per round	tok/s
  RTN (ModelOpt's recipe)	12.60 ± 0.28	2.30 ± 0.05	182.4 ± 3.1
  this GPTQ pack	12.60 ± 0.07	2.38 ± 0.06	188.6 ± 5.7

  The paired difference per round is -0.003 ± 0.32 ms. Tokens per round depend on how many MTP drafts each
  answer's text lets through (64-79% accepted here); three pairs cannot tell their difference from the answers' own
  spread.

Run it

Build or download Strata NVFP4 (Windows, NVIDIA driver 580+; see its README
for the requirements: RAM, pagefile, large pages). Then:

hf download Maximilian228/Huihui-Qwen3.8-Flash-Next-abliterated-NVFP4-GPTQ-Strata --local-dir models\huihui-gptq

One-shot (prompt.txt holds token ids):

strata.exe --pack models\huihui-gptq\pack ^
  --native models\huihui-gptq\huihui-nvfp4-dense.gguf --native-dense-gguf models\huihui-gptq\huihui-nvfp4-dense.gguf ^
  --ple-gguf models\huihui-gptq\ple-fp8.gguf --embd-gguf models\huihui-gptq\token-embd-bf16.gguf ^
  --mtp models\huihui-gptq\mtp --spec 6 --spec-min-p 0.7 --prefill auto ^
  --expert-profile data\expert-profile.bin --expert-cache auto ^
  --max-context 262144 --kv int8 --tokens-file prompt.txt --max-new 512 --stop-eos

--max-context 262144 fits a 32 GB card; with less VRAM lower it (the fork's README and setup use these):

data\expert-profile.bin ships with the engine. As a server (OpenAI and Anthropic APIs), put the same arguments in
the release's config\strata-nvfp4.json ("args") and point "tokenizer" at models/huihui-gptq/pack/tokenizer.
Pictures need the image encoder (mmproj-f32.gguf, not in this repo: the release's prepare-model.cmd builds
it from OrcaRouter's checkpoint, whose vision tower is byte-identical to huihui's, or convert_hf_to_gguf.py --mmproj
on the BF16 checkpoint); point the config's "vision" entry's "model" at huihui-nvfp4-dense.gguf (the encoder
reads only its vocabulary). Without the encoder, drop --vision from the arguments.

Already have the OrcaRouter NVFP4 repository? Its ple-fp8.gguf is this one, byte for byte; skip it with
--exclude ple-fp8.gguf and point --ple-gguf at yours. Its token-embd-bf16.gguf and mtp/ are OrcaRouter's
edited ones and do not belong to this model.

Reproduce

With the fork's tools and the BF16 checkpoint (hf download huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated, 361 GB):

1. The dense GGUF: llama.cpp's convert_hf_to_gguf.py with tools/nvfp4_convert.py's type policy, the routed
   experts left out; pack/dense.bin, pack/index.txt and pack/tokenizer/ from it with tools/iq_pack.py's
   index_standalone and tools/strata_tokenizer.py (every float tensor in the engine's form; the script's CLI
   wants a GGUF with experts); ple-fp8.gguf with tools/ple_fp8_pack.py,
   token-embd-bf16.gguf with tools/embd_bf16_pack.py, mtp/ with tools/mtp_extract.py -> mtp_pack.py --experts q2_0 -> mtp_rt.py (plus the engine's data/draft_vocab.bin).
2. A calibration pack by plain rounding, then the dump and GPTQ:

echo {"default": {"gu": "nvfp4", "d": "nvfp4"}} > calib\plan_nvfp4.json
.venv\Scripts\python tools\requant.py pack --bf16 models\huihui-bf16 --plan calib\plan_nvfp4.json --method rtn ^
  --base packs\huihui-dense --out packs\huihui-nvfp4-rtn
set STRATA_DUMP_MOE_INPUT=calib\d
set STRATA_DUMP_MOE_LAYER=all
strata.exe <the arguments above, --pack packs\huihui-nvfp4-rtn> --tokens-file data\requant_calib\calib_uk.txt --max-new 1   (and en, code, chat)
.venv\Scripts\python tools\requant.py analyze --bf16 models\huihui-bf16 --calib calib\d --methods rtn,gptq --out calib\errors.jsonl
.venv\Scripts\python tools\requant.py pack --bf16 models\huihui-bf16 --calib calib\d --plan calib\plan_nvfp4.json ^
  --method gptq --base packs\huihui-nvfp4-rtn --out packs\huihui-nvfp4-gptq

(packs\huihui-dense is the folder of step 1: dense.bin, index.txt, tokenizer/.) The packs here were built
layer group by layer group with the same requant.pack_layer calls, so the GPU was free between groups.

License

The Qwen Community License 1.0, as Qwen/Qwen3.8-Flash-Next and
huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated ship it (the same LICENSE file). Strata NVFP4 itself is MIT.