license: other
license_name: qwen-community-1.0
license_link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE
base_model: orcarouter/Qwen3.8-Flash-Next-Uncensored
base_model_relation: quantized
tags:
- exl3
- exllamav3
- qwen3.8
- flash-next
- abliterated
- uncensored
- mtp
- moe
- quantized
language: - en
- fr
- zh
pipeline_tag: image-text-to-text
Qwen3.8-Flash-Next-Uncensored — EXL3 3.05 bpw
EXL3 quantization of
orcarouter/Qwen3.8-Flash-Next-Uncensored,
an abliterated (refusal-removed) build of Qwen/Qwen3.8-Flash-Next.
Same parameters as turboderp's official 3.05bpw_h5_ng5 quant.
| Size on disk | 68 GB |
| Language model | 3.05 bpw |
lm_head |
5 bpw |
| MTP layers | 3 bpw |
| Vision tower | 5 bpw |
Also available: 4.05 bpw (102 GB).
Config
model:
model_name: Qwen3.8-Flash-Next-Uncensored-exl3-3bpw
cache_size: 32768
cache_mode: FP16
ngram_ram: true
tensor_parallel: false # required
draft_model:
draft_mode: mtp
tensor_parallel must be false. Otherwise:NotImplementedError: Tensor-parallel is not currently implemented for Qwen4ExpForConditionalGeneration. Autosplit works.
Cache quantization needs ExLlamaV3 ≥ 1.4.8. Below that, cache_mode
other than FP16 is rejected with QSA attention currently supports only the fp16 cache layer.
ngram_ram: true loads the n-gram table into system RAM instead of
streaming it from disk per forward.
MTP is preserved; TabbyAPI logs Using main model MTP component for drafting on load.
Conversion
python convert.py \
-i Qwen3.8-Flash-Next-Uncensored \
-o Qwen3.8-Flash-Next-Uncensored-exl3-3bpw \
-w /tmp/exl3-work \
-b 3.05 -hb 5 -mb 3 -vb 5 \
-d 0,1,2,3 -v
Default calibration corpus, 250 rows x 2048 columns.
Measurements
4x RTX 4000 Ada (20 GiB, sm_89), PCIe, no NVLink, 2x Xeon Gold 5318Y.
ExLlamaV3 1.5.0 + TabbyAPI, autosplit, MTP on, FP16 cache, 32k context.
AIPerf, streaming, ignore_eos, seed 42, with warmup.
ISL 1000 / OSL 500 / concurrency 4 / 20 requests:
| Metric | value |
|---|---|
| Request latency | 27,219 ms |
| Inter-token latency | 49.7 ms |
| Time to first token | 2,406 ms |
| Output throughput | 70 tok/s |
| Active prefill | 636 tok/s |
Single stream via chat.py, code prompt: 78 tok/s generation, 436 tok/s
prefill, 67% MTP acceptance.
20 requests per run, so gaps under ~5% are noise.
Quality
Not benchmarked at 3.05 bpw. Abliteration figures are orcarouter's, measured
on the BF16 source, not on this quant.
Safety
Safety alignment has been substantially removed via abliteration. It will
comply with requests the original Qwen3.8-Flash-Next refuses, and has no
meaningful built-in guardrails. Add your own moderation layer before any
deployment.
License
Qwen Community License 1.0, inherited from Qwen/Qwen3.8-Flash-Next.