license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/LICENSE
base_model: huihui-ai/Huihui-Qwen3.8-27B-abliterated
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: ninfer
language:
- en
- zh
tags: - qwen3_5
- qwen3.8
- ninfer
- nvfp4
- fp8
- mixed-precision
- blackwell
- rtx-5090
- sm120
- multimodal
- vision
- mtp
- reasoning
- abliterated
Huihui Qwen3.8-27B Abliterated — NInfer Mixed FP8/NVFP4
A complete NInfer artifact derived from huihui-ai/Huihui-Qwen3.8-27B-abliterated revision d42ca8978c5a66e92c3446d46e8adfe03ef692ff.
The artifact combines NInfer's fixed FP8 + NVFP4 Qwen3.8 allocation with same-source Huihui BF16 norms, embeddings, vision tower, and MTP tensors. No official Qwen or unrelated behavioral checkpoint supplies model weights.
Download and run
hf download lyf/Qwen3.8-27B-Huihui-Abliterated-NInfer-NVFP4 qwen3_8_27b_nvfp4.ninfer --local-dir ./huihui-ninfer
git clone https://github.com/Neroued/ninfer.git
cd ninfer
git checkout a99407c63fc5bbd25d9fb597cbb8ab352bdb01ef
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120a
cmake --build build -j"$(nproc)"
./build/apps/ninfer ../huihui-ninfer/qwen3_8_27b_nvfp4.ninfer --prompt "Explain FP8 and NVFP4 briefly." --max-new 128 --max-context 8192 --kv-dtype int8 --no-thinking
OpenAI-compatible server
./build/apps/ninfer-serve ../huihui-ninfer/qwen3_8_27b_nvfp4.ninfer --host 127.0.0.1 --port 8000 --model-id qwen38-huihui-ninfer-nvfp4 --max-context 204800 --kv-capacity 204800 --max-concurrency 1 --prefill-chunk 4096 --kv-dtype int8 --spec mtp --draft-tokens 3 --default-max-tokens 16384 --preserve-thinking
This 204.8K server profile was startup- and API-smoke-tested on one RTX 5090. Long-context quality and maximum sustained prompt length remain workload-dependent.
Mixed allocation
| Matrices | Format |
|---|---|
| Full attention and Gated DeltaNet projections | FP8 E4M3, per-output-row BF16 scales |
lm_head and layers 56–63 MLP |
FP8 E4M3 |
| Layers 0–55 MLP | NVFP4, group size 16 |
| Norms, GDN state, embeddings, vision, MTP | Same-source Huihui tensors |
NInfer source preflight passed with:
FP8 source matrices: 233
NVFP4 source matrices: 168
source fields: 1587
F8_E4M3: 401
BF16: 682
U8: 168
F32: 336
NINF_PREFLIGHT_OK
The FP8 matrices are deterministic row-scaled E4M3 exports from Huihui BF16. The NVFP4 matrices come from the separately calibrated Huihui ModelOpt NVFP4 checkpoint (CNN/DailyMail 3.0.0, 20 × 8192 tokens).
Artifact inventory
NInfer commit: a99407c63fc5bbd25d9fb597cbb8ab352bdb01ef
model_id: qwen3.8-27b
weights_id: nvfp4
objects: 1124 (1118 tensors, 6 resources)
artifact bytes: 21,492,695,040
SHA256: f21f308d3b23ccd627071cd015e413db08deee4356643900518e2b251750fdc2
Key formats:
BF16: 534
FP32: 208
FP8 row-scaled: 146
NVFP4: 112
Runtime validation
Hardware: RTX 5090 / SM120, driver 610.43.02, CUDA 13.1, 450 W cap.
Text smoke test:
prompt tokens: 27
generated tokens: 114
prefill: 792.69 tok/s
decode: 74.83 tok/s
overall: 73.83 tok/s
GPU weights: 18.98 GiB
Real-image test passed. For the test image, the model correctly identified a red square in the upper-left and a blue circle in the lower-right. Vision decode measured 74.82 tok/s with 19.25 GiB of GPU weights loaded.
A benign ablation/refusal-vector research prompt also received a direct technical answer. This is a smoke test of behavioral continuity, not a comprehensive behavioral benchmark.
Included evidence
qwen3_8_27b_nvfp4.ninfer.conversion.jsonninfer-nvfp4-preflight.logartifact-inspect.log- text, vision, and behavior outputs/logs
BUILD_MANIFEST.json,VALIDATION_REPORT.json,SHA256SUMS
Intended use
This is an abliterated behavioral derivative intended for model research and local inference. Users are responsible for downstream use and applicable policies.