library_name: openvino
license: apache-2.0
base_model:

• guglxni/Qwen3.5-9B-abliterated-DFlash
• lukey03/Qwen3.5-9B-abliterated
• z-lab/Qwen3.5-9B-DFlash
  pipeline_tag: text-generation
  tags:
• openvino
• int4
• dflash
• speculative-decoding
• draft-model
• block-diffusion
• abliterated
• uncensored
• qwen
• intel
• arc

Qwen3.5-9B-abliterated-DFlash int8 OpenVINO IR

OpenVINO IR conversion of
guglxni/Qwen3.5-9B-abliterated-DFlash,
an abliterated DFlash speculative-decoding draft for the Qwen3.5-9B family.

Sibling builds: int4 ·
BF16

What this is

A DFlash draft model — it is not run standalone. It attaches to a Qwen3.5-9B
target and proposes candidate tokens that the target verifies. It must be paired
with a target of matching geometry (see Pairing below).

Conversion details

The upstream repo ships custom modeling code (dflash.py) whose auto_map
points at itself, which prevents optimum-cli from exporting it — the dummy-input
check fails because that forward takes target_hidden and does not accept the
hidden_states/inputs_embeds contract optimum's generators produce.

The conversion therefore loads optimum's patched Qwen3DFlashDraftModel directly
and subclasses it to alias inputs_embeds onto noise_embedding:

from optimum.exporters.openvino import model_patcher
from optimum.exporters.openvino.model_configs import Qwen3OpenVINOConfig
from optimum.exporters.openvino.convert import export as ov_export

Base = model_patcher.Qwen3DFlashDraftModel

class DFlashDraft(Base):
    def forward(self, inputs_embeds=None, hidden_states=None, position_ids=None,
                attention_mask=None, past_key_values=None, use_cache=None,
                cache_position=None, noise_embedding=None, **kwargs):
        if noise_embedding is None:
            noise_embedding = inputs_embeds
        return super().forward(
            position_ids=position_ids, attention_mask=attention_mask,
            noise_embedding=noise_embedding, hidden_states=hidden_states,
            past_key_values=past_key_values, use_cache=use_cache,
            cache_position=cache_position, **kwargs)

cfg = AutoConfig.from_pretrained(SRC, trust_remote_code=False)
cfg.tie_word_embeddings = False
cfg.dflash = True
model = DFlashDraft(cfg)
model.load_state_dict(load_file(f"{SRC}/model.safetensors"), strict=False)

ov_export(model=model,
          config=Qwen3OpenVINOConfig(cfg, task="text-generation"),
          output="openvino_model.xml")

Note task="text-generation" — the internal config class rejects the CLI's
text-generation-with-past spelling.

The resulting IR was verified to match the upstream drafter's input contract:
inputs_embeds [?,?,4096], hidden_states [?,?,20480], last_hidden_state out.

Serving

Attach the draft to a target with --draft_model_path. DFlash requires prefix
caching to be disabled:

ovms --configure \
  --model_path <target model dir> \
  --task text_generation \
  --draft_model_path <this model dir> \
  --enable_prefix_caching false

ovms --config_path <target dir>/ovms_config.json --rest_port 11436

A successful pairing logs Draft model strategy: DFlash followed by
state: AVAILABLE.

Pairing — read this before benchmarking

This draft was trained against an abliterated target
(lukey03/Qwen3.5-9B-abliterated),
not vanilla Qwen3.5-9B. It also uses a different layer geometry from
z-lab/Qwen3.5-9B-DFlash:

Pairing this draft with vanilla Qwen3.5-9B loads and generates correctly, but
produces no speedup — measured 6.3 tok/s against 23.6 tok/s for the z-lab draft
on the same target. The draft predicts from hidden states it was not trained on,
so its proposals are rejected. Pair it with the abliterated target it was built
for.

Attribution

• Abliterated draft: guglxni/Qwen3.5-9B-abliterated-DFlash (Apache-2.0)
• Abliterated target: lukey03/Qwen3.5-9B-abliterated (Apache-2.0)
• Upstream DFlash draft: z-lab/Qwen3.5-9B-DFlash
• DFlash paper: arXiv:2602.06036
• Conversion: OpenVINO IR via optimum-intel + NNCF