license: other
license_name: qwen-research-use
base_model:
- RadixArk/Qwen3.8-27B-NVFP4
- theJuice85/Qwen3.8-27B-Abliterated-LoRA
base_model_relation: finetune
base_model_size: - 27B
library_name: safetensors
pipeline_tag: image-text-to-text
tags: - nvfp4
- compressed-tensors
- mtp
- multimodal
- uncensored
- abliteration
- speculative-decoding
- sglang
- vllm
- llama.cpp
- gb10
- qwen
language: en
Qwen3.8-27B Uncensored — NVFP4 + MTP (merged)
The merged serving artifact of a refusal-removal LoRA that was trained directly on a quantized GGUF — no dequantization — then folded back into the RadixArk/Qwen3.8-27B-NVFP4 checkpoint tree. One artifact, 2,400 tensors, 20.6 GB: vision, native MTP speculative decode, and the uncensored behavior, on vLLM, sglang, or (as a GGUF) llama.cpp. Measured end-to-end on an NVIDIA GB10 (DGX Spark, 121 GB unified memory).
| this repo | GGUF twin | |
|---|---|---|
| Format | compressed-tensors NVFP4 + FP8 scales | Q4_K_XL, same merged LoRA |
| Stack | vLLM / sglang | llama.cpp / bolt router |
| Weights | /tree/main (3 shards, 20.6 GB) |
Qwen3.8-27B-t26-Q4_K_XL.gguf |
Provenance — the full GGUF-space loop
- Train on the quant. heretic-gguf runs Heretic-style directional ablation ("obliteration") directly on a quantized GGUF of Qwen3.8-27B with llama.cpp as the runtime and Optuna as the search. Trial t26: refusal directions located and removed per-expert on the fused MoE tensors; the base file is never modified. → theJuice85/Qwen3.8-27B-Abliterated-LoRA
- Merge once. The adapter's low-rank directions were folded into every affected weight of the RadixArk NVFP4 tree (block-scale requantized after merge), restoring the vision tower and the trained MTP head.
- Serve anywhere. This tree runs on vLLM and sglang; the same merge also exists as a Q4_K_XL GGUF for llama.cpp — identical behavior, same test battery.
This is the point of the exercise: GGUF-space training is a full path now — train on the quant, merge once, serve on any engine.
Results — 172-prompt refusal battery
Semantic judge (Laya), McNemar exact test paired against the un-ablated NVFP4 base, same prompts both arms:
| NVFP4 base | this merged checkpoint | |
|---|---|---|
| Over-refusals (charged prompts) | 33 | ≤ base, McNemar p ≥ 0.50 |
| Newly introduced refusals | — | 0 |
| Gibberish ("salad") responses | 0 | 0 |
Behavior survived the entire quant → LoRA → merge → requant round trip: no regression flips, no collateral damage.
Performance — GB10 (DGX Spark, 121 GB unified memory)
Cold prefill = cache-busted 11.5k-token prompt (no prefix/radix cache hits). Decode = output tokens/s.
| Stack | Cold 11.5k prefill | Decode s1 | Decode c=8 | Spec decode |
|---|---|---|---|---|
| sglang NVFP4 (this repo) | 2,363 tok/s | 16.5 tok/s | — | native MTP, accept-len 2.3–2.85 |
| vLLM NVFP4 (this repo) | 2,267 tok/s | 17.7 tok/s | 99.6 tok/s | MTP-3, 57% acceptance |
| llama.cpp Q4_K_XL GGUF twin | 781 tok/s | 22–33 tok/s | — | ngram + MTP drafts |
Hybrid-GDN gotcha (the 2.6x one): on sglang, --mamba-ssm-dtype bfloat16 instead of the fp32 default was a 2.6x prefill swing on unified memory. The mamba state pins are not optional for Gated-DeltaNet hybrids.
Coexistence caveat: the two NVFP4 engines above ran alongside a llama.cpp tenant on the same box at --gpu-memory-utilization 0.55 (vLLM) / equivalent sglang fraction. On a dedicated GPU, raise the fraction accordingly.
1. Serve on vLLM (verified: vision + MTP-3 spec decode)
docker run -d --name qwen38-uncensored --device nvidia.com/gpu=all \
--shm-size 16g --ipc=host -p 30000:8000 \
-v /path/to/Qwen3.8-27B-Uncensored-NVFP4-MTP:/models:ro \
vllm/vllm-openai:qwen38-flash-next \
/models --served-model-name qwen38-uncensored \
--host 0.0.0.0 --max-model-len 16384 \
--gpu-memory-utilization 0.85 --trust-remote-code \
--limit-mm-per-prompt '{"image":4}' \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
MTP-3 acceptance measured 57% (1,655 accepted / 2,898 draft tokens) → ≈ +50% decode throughput vs no spec decode.
2. Serve on sglang (verified: vision + native MTP)
python -m sglang.launch_server \
--model-path theJuice85/Qwen3.8-27B-Uncensored-NVFP4-MTP \
--quantization compressed-tensors \
--mamba-ssm-dtype bfloat16 \
--mamba-full-memory-ratio 0.4 \
--chunked-prefill-size 8192 \
--max-running-requests 2 \
--disable-radix-cache \
--mem-fraction-static 0.30 \
--speculative-algorithm NEXTN --speculative-num-steps 1 \
--speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
--attention-backend triton --port 30001
Native MTP spec decode runs at accept-length 2.3–2.85 with vision verified through the same server. On a dedicated card raise --mem-fraction-static; --mamba-ssm-dtype bfloat16 is required for hybrid-GDN prefill speed (see gotcha above).
3. llama.cpp (GGUF twin)
Same merge, Q4_K_XL: Qwen3.8-27B-t26-Q4_K_XL.gguf in theJuice85/Qwen3.8-27B-Abliterated-LoRA. Serve with ngram speculative decoding (--spec-type draft-mtp --ngram-size 3) for the 22–33 tok/s single-stream numbers above.
Files
model-0000{1..3}-of-00003.safetensors— NVFP4 weights + FP8 weight-scale tensors (compressed-tensors)config.json— multimodal arch (vision + text + MTP head),Qwen3_5...ConditionalGenerationhf_quant_config.json— NVFP4 quant descriptors- tokenizer / chat template / vision + video preprocessor configs
License
Qwen Research license terms apply (inherited from the upstream base model).
