library_name: openvino
license: apache-2.0
base_model:
- lukey03/Qwen3.5-9B-abliterated
pipeline_tag: text-generation
tags: - openvino
- int4
- abliterated
- uncensored
- qwen
- intel
- arc
Qwen3.5-9B-abliterated int4 OpenVINO IR
OpenVINO IR conversion of
lukey03/Qwen3.5-9B-abliterated,
a full-layer abliteration (layers 0-31, scale 1.0) of Qwen3.5-9B.
Sibling build: int8
Details
| Property | Value |
|---|---|
| Architecture | Qwen3_5ForCausalLM, 32 layers, vocab 248320 |
| Format | OpenVINO IR, int4 asymmetric group-128 |
| Compression | 248/275 layers int4_asym g128, remainder int8 |
| Repository size | 5.3 GB |
Conversion notes
This architecture has a different transformers ceiling than LFM models: the
export gate rejects anything above 5.2.*, so the conversion needs
pip install "transformers==5.2.0"
whereas LFM heretic models need 5.4.0. Check the gate error if you convert
several families in one environment — it names the version it wants.
Large-model conversion used the two-stage method: export fp16 IR first, then
compress the IR with NNCF, rather than one optimum-cli --weight-format int8
run. This keeps peak memory far below a single-pass export.
Benchmark
Single-stream, Intel Core Ultra 7 258V (Arc 130V/140V iGPU), 30 GB RAM,
OpenVINO Model Server 2026.4.0 on GPU:
| Configuration | tok/s | TTFT |
|---|---|---|
| int8, no draft | 8.2 | 0.157 s |
| int8, with abliterated DFlash draft | 7.5 | 0.193 s |
Pairing with the abliterated DFlash draft
blaj/Qwen3.5-9B-abliterated-DFlash-int4-ov
is the matching draft for this target. Measured against it, speculative decoding
was slower than no draft at all (7.5 vs 8.2 tok/s) — the draft's proposals are
rejected. It loads and runs correctly, but provides no speedup. See that repo's
card for the geometry comparison against z-lab/Qwen3.5-9B-DFlash.
Serving
PYTHONPATH=$OVMS_ROOT/lib/python ovms --config_path ./ovms_config.json --rest_port 11436
PYTHONPATH must point at the OVMS lib/python; without it the server exits
immediately after initializing the Python interpreter with no error.
Attribution
- Abliteration: lukey03/Qwen3.5-9B-abliterated (Apache-2.0)
- Conversion: OpenVINO IR via
optimum-intel+ NNCF