library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
tags:
- documentation
- diffusion-lm
- masked-diffusion
- gguf
- not-supported
Why there is no GGUF for the diffusion checkpoint
Documentation repository. It contains no weights, on purpose.
Short version
OpceanAI/Qwen3.5-0.8B-abliterated-diffusion is a masked diffusion language model. GGUF and llama.cpp describe causal, autoregressive inference. The two do not meet.
There is nothing to download here. Use the safetensors checkpoint:
from huggingface_hub import snapshot_download
path = snapshot_download("OpceanAI/Qwen3.5-0.8B-abliterated-diffusion")
Why the conversion does not work
A GGUF file stores tensors plus enough metadata for llama.cpp to rebuild a forward pass. For this model, the forward pass is not the one recorded in the weights.
| What the model needs | What GGUF can express |
|---|---|
| Bidirectional attention over the full sequence | Causal mask, or a fixed set of arch flags |
| Forward scan + gated reverse scan in each DeltaNet layer | A single causal recurrence |
rev_gate tensors injected into the layer forward |
Only named tensors with known roles |
Shifted prediction (j from hidden j-1) |
Standard next-token head |
mask_embed for the [MASK] token |
No concept of a mask token |
| Iterative parallel unmasking at inference | Token-by-token sampling |
The first four are Python-level surgery applied at load time by dqwen.core.patch_bidirectional. They are not weights, so they cannot be serialized into a GGUF.
What happens if you try anyway
Converting the checkpoint with convert_hf_to_gguf.py succeeds, because the tensors look like a normal Qwen3.5 text model. The output is misleading:
rev_gateandmask_embedare dropped. They are not in the standard tensor namespace.- Attention loads as causal.
- The model then generates as if it were autoregressive, but it was trained with bidirectional context.
The result is a causal model with the wrong inference geometry. It runs, and it produces degraded text. Shipping that file under a "GGUF" label would be a broken artifact, which is why this repository has none.
# This is what the conversion produces, and why we do not publish it
python convert_hf_to_gguf.py ./Qwen3.5-0.8B-abliterated-diffusion --outtype f16
# -> a causal GGUF with no rev_gate, no mask embedding, no diffusion sampler
What would have to exist first
Three pieces, none of which exist today:
- A GGUF metadata schema for hybrid bidirectional models.
llama.cpphas per-architecture flags, so an arch could declarebidirectional,deltanet_reverse, andnextn_shift. Nobody has written it. - A
rev_gatetensor type in the loader. It is a[hidden]fp32 vector per DeltaNet layer, which is easy to store, but the graph has to consume it. - A denoising sampler in the runtime. Parallel unmasking is a different decoding loop, not a sampling parameter.
The reference implementation for the first two exists in Python inside the model repository. Porting it to C++ is a project, not a conversion command.
Where the quants do exist
The autoregressive sibling is fully quantized, because it is a normal causal model:
OpceanAI/Qwen3.5-0.8B-abliterated-GGUF: F16, BF16, Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M, Q2_K, IQ4_XS, IQ3_M, IQ3_XXS, IQ2_M, IQ2_XXS, TQ1_0, TQ2_0.
How to run the diffusion model instead
pip install -U transformers accelerate huggingface_hub safetensors flash-linear-attention
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("OpceanAI/Qwen3.5-0.8B-abliterated-diffusion")
sys.path.insert(0, path)
from dqwen.hub import load_pretrained, generate
dlm, tok = load_pretrained(path)
print(generate(dlm, tok, "Explain why the sky is blue.", gen_len=96, thr=0.9))
Needs a GPU for practical speed. The model is 752M parameters in bf16, about 1.5 GB.