license: apache-2.0
language:
- en
pipeline_tag: text-generation
tags: - qwen
- qwen3.8
- int8
- qengine
- uncensored
- mtp
Qwen3.8-27B-Uncensored-INT8-QEngine
Community INT8 conversion of Qwen3.8-27B-Uncensored-BF16 for the QEngine inference runtime. This is a third-party conversion, not an official Alibaba Cloud/Qwen release.
Quantization and files
- Main weights: symmetric W8A16, group size 128; generated with
tools/quant_int8.py(search=0). - MTP head: original BF16 weights.
- KV scales: sourced from the local
Qwen3.8-27B-INT4checkpoint. - Includes model configuration, tokenizer, chat template, and the bundled Apache-2.0 license file.
model.safetensors: 30,766,875,432 bytes;model_mtp.safetensors: 849,400,424 bytes.
QENGINE_INT8.md records quantization provenance. The exact upstream Hugging Face repository ID and converter revision are not recorded in the supplied metadata, so no upstream repository link is asserted here.
Inference
Requires a QEngine build supporting this Qwen3.8 INT8 format and MTP head. Tested with ghcr.io/bpx03/qengine:1.3.5.2-vision, command --vision 1, and the OpenAI-compatible /v1/chat/completions API. These weights are QEngine-specific; Transformers/vLLM compatibility has not been tested and must not be assumed.
Server settings: QE_MAX_SEQS=4; QE_COMM=auto selects copy-engine prefill transfers; QE_DECODE_COMM=auto calibrates decode transport (this rig selected pinned host memory over P2P). Server sampling defaults when requests omit these fields: temperature 0.60, top-p 0.95, top-k 20, min-p 0.00.
curl http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"qwen3.8-27b-qengine","messages":[{"role":"user","content":"Explain quantization in one paragraph."}],"max_tokens":256}'
Benchmark
Measured 2026-11-07 with the club-3090 scripts/bench.sh; RUNS=3, WARMUPS=1. Sampler: temperature 0.6, top-p 0.95, top-k 20, min-p 0.0. Three RTX 3090 GPUs; QEngine-reported context 262,144 tokens, four served slots, FP8 E4M3 KV cache.
| Prompt shape | Wall throughput (mean) | Decode throughput (mean) | TTFT (mean) | Samples |
|---|---|---|---|---|
| Narrative, 1,000-token cap | 134.00 tok/s | 135.28 tok/s | 70 ms | 3/3 usable; decode CV 1.2% |
| Code, 800-token cap | 171.74 tok/s | 175.15 tok/s | 71 ms | 3/3 usable; decode CV 3.5% |
| Prefill, ~10K input tokens | — | 3,574.80 prompt tok/s | 2,833 ms | 3; CV 2.3% |
| Prefill, ~90K input tokens | — | 3,199.55 prompt tok/s | 28,128 ms | 1; directional only |
Power caveat: the full rig report detected all three GPUs capped at 250 W (GPU 0: 71% of 350 W default; GPU 1: 59% of 420 W; GPU 2: 71% of 350 W). Treat throughput above as power-limited, not as a stock-power baseline.
Separate concurrency probe: four simultaneous streams, approximately 16K input and 256 generated tokens per stream, five rounds, greedy sampling (temperature 0). PASS: 4/4 completed without errors in every round; 79.03 decode tok/s per stream (steady), 9,059 ms median concurrent TTFT, 17,991 ms TTFT p95, 36 MiB post-warm VRAM growth. Greedy concurrency results are not directly comparable to the benchmark sampler.
The full scripts/report.sh --full rig report passed all five stages: verify-full.sh, verify-stress.sh, soak-test.sh, bench.sh, and bench-agentic.sh.
Limitations
- Results apply to this QEngine build and three-GPU rig; GPU power caps above materially limit comparison with stock-power runs.
- The 90K prefill result has one sample. No quality evaluation or BF16 comparison was performed.
- The bundled license identifies Apache License 2.0 and copyright Alibaba Cloud. Confirm the exact source checkpoint repository and license before redistribution; preserve all upstream notices.
- Follow upstream acceptable-use and safety requirements. “Uncensored” is a model label, not a safety property.