license: apache-2.0
base_model: orcarouter/Nex-N2.5-mini-Uncensored
tags:
- auto-round
- w4a16
- int4
- gptq
- vllm
- qwen3_5_moe
- uncensored
- abliterated
pipeline_tag: text-generation
Nex-N2.5-mini-Uncensored-W4A16-AutoRound
W4A16 (INT4 weights, group size 128, symmetric, BF16 activations) quantization of
orcarouter/Nex-N2.5-mini-Uncensored
(the abliterated Nex-N2.5-mini, a Qwen3.5-MoE 35B-A3B vision-language model) inauto_gptq format, loadable by vLLM (Marlin / XPU) and SGLang.
Why this exists
orcarouter publishes the Uncensored weights as BF16 (65 GiB), FP8 (34 GiB, Hopper+),
NVFP4 (22 GiB, Blackwell only), MLX (Apple) and GGUF (llama.cpp). None of those loads
in vLLM on a 32 GiB non-Blackwell card. [quant-mind/Nex-N2.5-mini-W4A16-AutoRound]
(https://huggingface.co/quant-mind/Nex-N2.5-mini-W4A16-AutoRound) filled that gap for
the STOCK model; this repository applies the same published recipe to the Uncensored
weights, so the two differ in base weights only. Built for the murakumo fleet's
Intel Arc Pro B70 (32 GiB) head, where vLLM XPU + XPU graph measured ~1,128 tok/s
aggregate on the stock W4A16 checkpoint vs ~137 tok/s for llama.cpp IQ4_XS on the same
card (root ADR-2609181615, com-junkawasaki/root).
Recipe (identical to quant-mind's config.json)
- Intel AutoRound 0.15.1,
--scheme W4A16 --group_size 128(sym),--iters 200,--nsamples 128 --seqlen 2048, calibrationNeelNanda/pile-10k,--low_gpu_mem_usage - Kept in BF16:
mlp.gate(router),mlp.shared_expert_gate,mtp,lm_head,
embeddings, the visual tower - Quantized:
linear_attn.*_proj,self_attn.{q,k,v,o}_proj, all 256mlp.experts.*,mlp.shared_expert.* - Export format:
auto_gptq - Built on Modal (H100) by the job
tools/modal-quant/nex_n25_mini_uncensored_w4a16.py
in kotoba-lang/murakumo.
Use
vllm serve com-kotobalabs/Nex-N2.5-mini-Uncensored-W4A16-AutoRound --reasoning-parser qwen3 --max-model-len 65536 --gpu-memory-utilization 0.90
The model thinks before answering; without --reasoning-parser qwen3 the reasoning
leaks into content.
Caveats
- Abliterated / uncensored derivative: it will not refuse. Deploy behind your own policy
layer; the murakumo gateway runs it only behind governed organisms. - Quality vs the BF16 source is not evaluated here beyond a coherence probe; the
quantization error profile is the one AutoRound publishes for this recipe. - License follows the base: Apache-2.0 (nex-agi/Nex-N2.5-mini) as relicensed by orcarouter.