license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
base_model:

• huihui-ai/Huihui-Qwen3.8-27B-abliterated
• gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090
  tags:
• qwen3.8
• abliterated
• nvfp4
• modelopt
• sglang
• dspark
• rtx-5090

Qwen3.8-27B NVFP4 RTX5090 — Huihui Abliterated

The goal of this release was simple: take the original BF16 Huihui abliterated weights and quantize them with the same practical NVFP4 layout used by Gittensor's RTX 5090 release.

This is an independent derivative, not an official Huihui, Gittensor, Qwen, NVIDIA, or SGLang release.

Provenance

What was done

1. Downloaded the pinned raw BF16 Huihui checkpoint: 18 shards and 55,563,006,216 weight-file bytes.
2. Removed the unchanged native MTP head from the deployment view; production uses either normal autoregressive decoding or an external drafter.
3. Ran NVIDIA ModelOpt PTQ with Gittensor's Qwen3.8 all-linear NVFP4 recipe:
   • NVFP4 W4A4, group size 16;
   • 128 image-text calibration samples;
   • calibration sequence length 512;
   • vision, embeddings, and Gated-DeltaNet conv1d, in_proj_a, and in_proj_b kept out of NVFP4.
4. Normalized the output to Gittensor's tensor names, shapes, dtypes, shard map, and quantization metadata.
5. Used Gittensor's NVFP4 lm_head byte-for-byte. Huihui states that lm_head was not modified, so this preserves the qualified Gittensor head rather than independently recalibrating it.
6. Used Gittensor's chat template and tokenizer metadata so stock and abliterated deployments have the same serving format.
7. Verified the finished artifact against the pinned Gittensor layout.

All model tensors other than the explicitly described lm_head come from a fresh PTQ export of the pinned Huihui source.

Huihui's card says layers 18–51 were ablated. Direct tensor comparison found changed zero-based mlp.down_proj layers 17–51. Both observations are recorded without trying to reinterpret Huihui's numbering convention.

Artifact checks

• 2 safetensor shards
• 2,387 tensors
• 17,915,815,528 tensor payload bytes
• 17,916,112,584 safetensor file bytes
• no MTP tensors
• exact Gittensor tensor layout and quantization exclusions
• SHA256 hashes in SHA256SUMS
• machine-readable details in PROVENANCE.json

Run with SGLang: native 262k, no speculation

This is the cleanest way to run the model. The command below was qualified with one RTX 5090 32 GB and one request:

MODEL_DIR=/path/to/Qwen3.8-27B-NVFP4-RTX5090-Gittensor-Abliterated-Huihui

docker run --rm --gpus all --ipc=host --shm-size=32g \
  -p 127.0.0.1:8000:30000 \
  -v "$MODEL_DIR:/model:ro" \
  --entrypoint python3 \
  lmsysorg/sglang@sha256:febfb971c7352570fc445c466ebd6ffc9d896024958e544a60f2137fd85856b1 \
  -m sglang.launch_server \
    --model-path /model \
    --served-model-name qwen3.8-27b-abliterated-nvfp4 \
    --trust-remote-code --tp-size 1 \
    --host 0.0.0.0 --port 30000 \
    --context-length 262144 \
    --kv-cache-dtype fp8_e4m3 \
    --attention-backend flashinfer \
    --chunked-prefill-size 1024 \
    --disable-prefill-cuda-graph \
    --cuda-graph-max-bs-decode 1 \
    --mamba-radix-cache-strategy extra_buffer_lazy \
    --mamba-ssm-dtype bfloat16 \
    --max-mamba-cache-size 5 \
    --mem-fraction-static 0.93 \
    --max-running-requests 1 \
    --reasoning-parser qwen3 \
    --default-chat-template-kwargs '{"reasoning_effort":"medium","preserve_thinking":true}' \
    --tool-call-parser qwen3_coder \
    --mm-feature-transport cpu \
    --language-only \
    --enable-metrics

On WSL with NVIDIA Container Toolkit CDI, replace --gpus all with --device nvidia.com/gpu=all if required.

Faster agent setup: SGLang + DSpark

For faster decoding, download the separate Gittensor DSpark checkpoint linked above and add these flags to the same SGLang command:

-v /path/to/Qwen3.8-27B-DSpark-NVFP4:/model_dspark:ro

--context-length 202000 \
--speculative-algorithm DSPARK \
--speculative-draft-model-path /model_dspark \
--speculative-dspark-block-size 7 \
--speculative-draft-model-quantization modelopt_fp4 \
--speculative-draft-attention-backend flashinfer

Our qualified single-agent client budget is:

shared context: 202,000
max output:      32,000
prompt + tools:  170,000
physical pool:   203,125

It is 170k, not 180k, because output, reasoning, prompts, tool schemas, and tool results all share the same 202k sequence budget.

The qualified drafter is DSpark, not DFlash. DFlash was tested separately but was not the recommended production configuration.

Observed performance

On one custom 4,096-step agent coding benchmark:

These are single runs on a custom benchmark, not a general leaderboard claim.

Limitations

• Abliteration reduces refusal behavior and safety filtering. Treat outputs as untrusted.
• The 262,144-token value is the target-only architectural/qualified serving window. External speculation uses extra memory and reduces the practical shared window on a 32 GB card.
• Speculative decoding accelerates decode, not cold prefill.
• The qualified deployment was language-only; multimodal behavior was not evaluated here.
• Do not expose the unauthenticated API publicly.

License

Apache License 2.0. See LICENSE.

Credits: Qwen team; Huihui AI for the abliterated source; Gittensor Model Hub for the NVFP4 recipe/layout and DSpark; NVIDIA for Model Optimizer. Independent quantization, validation, and packaging by heswithme.