license: apache-2.0
base_model: orcarouter/Qwen3.8-27B-Uncensored
base_model_relation: quantized
language:

• en
• zh
  pipeline_tag: image-text-to-text
  tags:
• qwen3.8
• qwen3_5
• nvfp4
• fp4
• compressed-tensors
• vllm
• blackwell
• rtx-5090
• uncensored
• abliterated
• quantized
• vision-language
• function-calling
• reasoning
• coding-agent

Qwen3.8-27B-Uncensored-NVFP4

NVFP4 (W4A4) quantization of
orcarouter/Qwen3.8-27B-Uncensored,
an abliterated fine-tune of Qwen/Qwen3.8-27B.
~19GB (down from ~56GB BF16) — built for single consumer Blackwell GPUs.
On an RTX 5090 (32GB) it serves with 160k context and room to spare.

• FP4 weights and FP4 activations on the transformer linear layers →
  native speedups on Blackwell (SM120: RTX 50-series, RTX PRO; SM100: B200)
• Kept in BF16: lm_head, the vision tower, and all gated-DeltaNet
  linear_attn layers (the hybrid architecture's linear-attention blocks)
• Vision, tool calling, and reasoning (<think>) all functional

Serving with vLLM

Verified config for a single RTX 5090 (32GB), tested on vLLM 0.27.1:

vllm serve preetpatel/Qwen3.8-27B-Uncensored-NVFP4 \
    --max-model-len 163840 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.87 \
    --max-num-seqs 8 \
    --max-num-batched-tokens 8192 \
    --limit-mm-per-prompt '{"image": 2, "video": 0}' \
    --mm-processor-kwargs '{"max_pixels": 2097152}' \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3

Notes from real-world testing on the 5090:

• Leave headroom. A tighter config (BF16 KV, 131k context,
  higher utilization) boots fine but OOMs mid-request: the gated-DeltaNet
  prefill kernel makes ~200MB transient allocations that the startup
  profiler does not fully account for. --gpu-memory-utilization 0.87
  leaves ~1.5GB of slack; PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
  helps against fragmentation.
• FP8 KV cache halves KV to ~32KiB/token. The hybrid architecture only
  keeps KV for 16 of 64 layers, so 160k context needs only ~5GB of KV.
• Tool calls use the Qwen3-Coder XML format
  (<function=...><parameter=...>) — --tool-call-parser qwen3_coder
  is required for structured tool_calls output; --reasoning-parser qwen3
  moves <think> content into reasoning_content.
• The model's generation_config.json supplies the recommended sampling
  defaults (temperature 1.0, top_p 0.95, top_k 20).

Measured results (RTX 5090, vLLM 0.27.1)

Also verified end-to-end as the backend of a coding agent
(pi): multi-turn tool loop, file
writes, and shell execution all behave correctly.

Known caveat

vLLM logs at load time:

In NVFP4 linear, the weight global scale is different for parallel layers
(e.g. q_proj, k_proj, v_proj).

This checkpoint stores an independent NVFP4 global scale per linear layer;
vLLM reconciles them when fusing q/k/v (and gate/up) GEMMs, which adds a
small one-time rounding cost on those layers. The functional testing above
(tool calling, deep-context retrieval, agent use) shows no observable
degradation, but exact-benchmark users should be aware. If future
llmcompressor releases expose fused-layer scale sharing, a v2 revision may
be published.

How it was made

One-shot RTN quantization with llmcompressor
0.13.0 (compressed-tensors 0.17.0):

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=["lm_head", "re:.*visual.*", "re:.*mtp.*"],
)

Calibration: 20 samples from ultrachat_200k (2048 max sequence length),
used only to set activation global scales — no training involved. The
exact recipe ships in this repo as recipe.yaml; the full quantization
and serving setup lives at
preetpatel/qwen3.8_27B-nvidia5090.

Attribution & license

Apache 2.0, inherited from the base model. All credit for the model
weights to Qwen (base model) and
orcarouter (abliterated fine-tune);
this repo only changes the numeric format. The base model is an
abliterated/uncensored variant intended by its authors for AI red-teaming
and research — apply your own judgment and safety measures downstream.