license: apache-2.0
base_model: DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: transformers
language:
- en
- zh
tags: - qwen3.6
- nvfp4
- w4a16
- compressed-tensors
- vllm
- quantized
- uncensored
- heretic
- creative-writing
- roleplaying
- vision-language
- mtp
Fable-Fusion 711 for vLLM (NVFP4A16)
This is the vLLM version of DavidAU's Qwen3.6-27B Fable-Fusion 711. The weights are FP4. Everything sensitive stays BF16. The repo is 28 GB instead of 56. It serves as an OpenAI-compatible endpoint on any NVIDIA GPU with about 32 GB of VRAM, Ampere or newer. Blackwell is not required.
| Scheme | NVFP4A16 weight-only, compressed-tensors format, native vLLM support |
| Source | quantized directly from the BF16 safetensors. No GGUF step in the lineage. |
| Kept in BF16 | all 48 linear-attention (DeltaNet) layers, the vision tower, the MTP drafter, lm_head |
| Size | 27.7 GB weights (BF16 parent: 55.6 GB) |
| VRAM | ~32 GB minimum; tested on 1x RTX PRO 6000 Blackwell (96 GB) |
| Context | up to 262,144 tokens, VRAM permitting |
| Vision | included, unquantized |
| MTP speculative decoding | verified: 1.56x decode speedup at draft depth 5 |
| FP8 KV cache | calibrated k_scale/v_scale included; verified with --kv-cache-dtype fp8 |
| License | apache-2.0, inherited from the parent |
Quickstart
vllm serve kkuspa/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-NVFP4A16 \
--max-model-len 32768 --gpu-memory-utilization 0.85
Tested on vLLM 0.26.0. The server selects the Marlin NVFP4 kernel and warns that the GPU lacks native FP4 compute. The warning is expected: activations are BF16 by design, so the FP4 tensor-core path does not apply.
For faster decoding, enable the built-in MTP drafter:
vllm serve kkuspa/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-NVFP4A16 --max-model-len 32768 --gpu-memory-utilization 0.85 --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":5}'
DavidAU tuned this model for specific sampler settings. They translate to the OpenAI API as follows:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
# Thinking mode (default template): creative work, reasoning
r = client.chat.completions.create(
model="kkuspa/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-NVFP4A16",
messages=[{"role": "user", "content": "Open a noir story set in a lighthouse."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
extra_body={"top_k": 20, "min_p": 0.0, "repetition_penalty": 1.0},
)
# Coding: temperature=0.6, same top_p and top_k
# Instruct mode: temperature=0.7, top_p=0.80, presence_penalty=1.5
The repo ships two chat templates. chat_template.jinja (thinking) is the default. For instruct-style responses without a thinking block, pass --chat-template pointing at chat_template-instruct.jinja at serve time.
This repo or the GGUF?
- llama.cpp, LM Studio, Ollama, one user on a consumer card: use DavidAU's GGUF repo.
- vLLM, concurrent requests, an OpenAI-compatible server, or 100k+ context: use this repo.
At the time of publication this is the only quant of Fable-Fusion 711 that stock vLLM can serve. The other safetensors quants of this model use a custom runtime format.
Fidelity
Measured with lm-evaluation-harness 0.4.12 (vLLM backend, 0-shot, batch auto) against the BF16 parent on the same hardware:
| Task | BF16 | NVFP4A16 | Delta |
|---|---|---|---|
| ARC-Challenge (acc_norm) | 0.6126 | 0.6203 | +0.008 |
| HellaSwag (acc_norm) | 0.8502 | 0.8490 | -0.001 |
| Winogrande (acc) | 0.7806 | 0.7814 | +0.001 |
Deltas sit inside the standard error on every task (ARC-Challenge stderr is 0.014). The quantization is a data-free RTN pass. No calibration dataset was used, so nothing was tuned toward any benchmark. Absolute scores depend on harness settings; the point of this table is the like-for-like comparison, both configs measured 0-shot on the same harness and hardware.
Speed
Single request, 512-token greedy generations, 1x RTX PRO 6000 Blackwell (96 GB), vLLM 0.26.0:
| Config | Decode speed |
|---|---|
| Standard decoding | 56.2 tok/s |
| MTP speculative, depth 5 | 87.4 tok/s (1.56x) |
Acceptance across the test: 996 of 3000 drafted tokens (about 1.7 extra tokens per draft window). Throughput under concurrent load will differ.
What was quantized
llm-compressor 0.12.0, one-shot:
QuantizationModifier(
targets="Linear",
scheme="NVFP4A16",
ignore=["lm_head", "re:.*visual.*", "re:.*linear_attn.*", "re:.*mtp.*"],
)
Only the Linear weights in the full-attention and MLP blocks carry FP4. The DeltaNet linear-attention layers, the vision tower, the MTP drafter, and lm_head keep BF16, because hybrid-attention layers and drafters degrade badly under 4-bit quantization. The exact recipe ships in recipe.yaml.
Packaging notes
- The parent repo ships no image-processor configs.
preprocessor_config.jsonandvideo_preprocessor_config.jsonhere come from Qwen/Qwen3.6-27B. - The 15 MTP drafter tensors live in
model-mtp-bf16.safetensorsand are wired intomodel.safetensors.index.json. Transformers drops them on load; they were re-extracted from the parent checkpoint so speculative decoding remains possible.
Content notice
The parent model is refusal-ablated ("Uncensored", "Heretic") and tuned for unconstrained creative writing. This quant changes none of that. It will produce content that aligned models refuse. You are responsible for how you deploy it.
Caveats
- The vision path loads but has not been benchmarked.
Lineage and attribution
Qwen/Qwen3.6-27B by the Qwen team, then the Fable-Fusion 711 multi-stage tune and merge by DavidAU with collaborators credited on the parent card, then this quantization. Thanks to all of them. Apache-2.0 throughout.