base_model: xtools-at/DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic
base_model_relation: quantized
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- compressed-tensors
- nvfp4
- quantized
- qwen3_5
DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4
Weight-only NVFP4A16 quantization of xtools-at/DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic, created on 2026-11-07. Source revision: ef863796ba0d5fb36aa9363bcdc6332387885f59.
This is a quantized Qwen3.5-9B derivative. The upstream model name describes its distillation lineage; this checkpoint is not the full DeepSeek-V4-Pro model.
Quantization
- Format: compressed-tensors
nvfp4-pack-quantized, FP4 E2M1 weights / BF16 activations. - Group size 16, FP8 E4M3 block scales and FP32 global scales. No activation FP4 quantization or calibration dataset.
- 128 language MLP and full-attention Linear modules quantized.
- Vision, linear attention, embeddings, lm_head and norms retained in BF16. All 632 unquantized tensors, including 333 vision tensors, verified bit-exact against the source.
- Weight files: 11,198,703,656 bytes in six shards (source: 18,819,627,488 bytes).
- LLM Compressor 0.11.0, compressed-tensors 0.16.0, Transformers 5.9.0, PyTorch 2.11.0+cu128.
- Exported using the Qwen3.5 NVFP4A16 recipe that excludes linear attention and vision. 81 non-quantized norm tensors were restored from the source after detecting BF16 rounding in the compressor's offset-norm conversion round trip.
- The source config declares an MTP layer, but the source checkpoint contains no MTP tensors. No MTP weights were added; speculative decoding was not tested.
See recipe.yaml, checksums.sha256, and the retained upstream model card.
Validation
The saved checkpoint was decompressed and reloaded for arithmetic and image smoke checks: 17+25 → 42, 9×9 → 81; a red-left / blue-right image was correctly described in Japanese. This image check used decompressed BF16 execution, not native quantized vision serving.
Native text serving was then verified with vLLM 0.33.0 (eadd44bb) using MarlinNvFp4LinearKernel and FLASH_ATTN. Both arithmetic checks passed. The native serving benchmark used --language-model-only; native vision serving remains untested.
Text throughput measured on one RTX PRO 2000 Blackwell 16 GB
GPU shared with the display. The resident TTS service was temporarily stopped. TP=1, all loaded text weights GPU-resident, no CPU offload. No speculative decoding. Results are medians of three measured runs after one warmup per condition.
- Prefill: 8,192 input tokens, 1 output token. Aggregate input tokens divided by the slowest TTFT, including scheduling and first-token overhead.
- Decode: 512 input tokens, 256 output tokens, EOS ignored. Aggregate generated tokens excluding each first token divided by the interval from the earliest first token to the last request completion. Per-request speed is the mean of individual stream speeds, then the median across runs; it is not aggregate divided by concurrency.
- Prefix caching disabled; max model length 16,384; max sequences 8; max batched tokens 1,024; GPU memory utilization 0.82; OMP_NUM_THREADS=4; CUDA graph capture sizes [1,2,4,8].
- Shared KV capacity reported by the server: 30,427 tokens. Long prefill requests include queueing; eight full 16,384-token contexts do not fit simultaneously.
| Concurrent requests | Aggregate prefill tok/s | Aggregate decode tok/s | Decode tok/s per request |
|---|---|---|---|
| 1 | 2512.6 | 29.35 | 29.35 |
| 2 | 2416.3 | 55.14 | 28.00 |
| 4 | 2381.8 | 106.66 | 28.02 |
| 8 | 2351.0 | 202.35 | 27.52 |
All 24 measured batches (90 requests) completed with the expected input/output token counts. See benchmark-summary.json for all three samples. These results describe this hardware and configuration, not a general maximum performance claim.
Tested text serving example
With vLLM 0.33.0 and a supported CUDA setup:
vllm serve sakamakismile/DeepSeek-V4-Pro-Qwen3.5-9B-UNCENSORED-heretic-NVFP4 \
--tensor-parallel-size 1 \
--language-model-only \
--max-model-len 16384 \
--max-num-seqs 8 \
--max-num-batched-tokens 1024 \
--gpu-memory-utilization 0.82 \
--no-enable-prefix-caching \
--reasoning-parser qwen3 \
--seed 42 \
--compilation-config '{"cudagraph_capture_sizes":[1,2,4,8]}'
Memory settings depend on other GPU users. The model itself took 9.65 GiB to load in the tested text-only configuration; runtime workspaces and KV cache require additional memory. The full checkpoint retains the BF16 vision weights.
License and attribution
Apache-2.0, following the upstream repository metadata. Credit to xtools-at for the source heretic release, Jackrong for the distilled model, and the Qwen team for Qwen3.5. Upstream training and benchmark claims are preserved separately in SOURCE_README.md and were not independently reproduced here.