library_name: transformers
license: apache-2.0
language:
- en
base_model: - Qwen/Qwen3.6-27B
pipeline_tag: image-text-to-text
tags: - text-generation-inference
- mtp
- llama-cpp
- abliterated
- nvfp4
- nvidia
- llama-cli
- speculative-decoding
- nvidia-blackwell
- rtx-pro-6000
- SM120
- GB202
- B200
- B300
- FP4
- blackwell
- qwen3_5
- Qwen3.5
- Qwen3.6
- Qwen3.8
- 27B
Qwen3.6-27B-abliterated-NVFP4-GGUF
Qwen3.6-27B from Alibaba's Qwen team is a 27B-parameter dense multimodal model with an integrated vision encoder, 64 layers, and a native 262K-token context window extendable to 1M+ via YaRN. It delivers strong agentic coding, repository-level reasoning, and frontend development capabilities, outperforming the previous Qwen3.5-397B-A17B flagship on SWE-bench Verified (77.2), Terminal-Bench (59.3), and SkillsBench (48.2). It also scores 87.8 on GPQA-Diamond and 66.1% on BFCL-V4 for tool calling. Hybrid thinking modes preserve reasoning across multi-turn workflows. It supports vLLM, SGLang, and Unsloth, with GGUF quantization enabling local deployment, and is released under Apache-2.0.
[!NOTE]
MTP — Speculative Decoding: Yes
[!NOTE]
Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.
Model Files
| File Name | Quant Type | File Size | File Link | Description |
|---|---|---|---|---|
| Qwen3.6-27B-abliterated-NVFP4.gguf | NVFP4 | 27.6 GB | Link | NVFP4 quantization. Highly optimized for NVIDIA Blackwell hardware accelerators (e.g., RTX Pro 6000, RTX 50-series, B200, B300), offering native hardware-accelerated FP4 Tensor Core speeds and a strong balance of size and performance for this 27B parameter model. |
| Qwen3.6-27B-abliterated-NVFP4.mmproj.gguf | mmproj | 931 MB | Link | Multimodal projection file. Used for vision/language models. |
NVFP4 Test Run
Tested Qwen3.6-27B-abliterated-NVFP4.gguf with vision support on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB VRAM, using llama-cli with full GPU offloading and MTP speculative decoding.
Performance Metrics
| Metric | Result |
|---|---|
| Average Prompt Processing Speed | 1,157.93 tokens/s |
| Average Generation Speed | 101.08 tokens/s |
| Average Wall Time per Image | 18.90 seconds |
| Total Wall Time (3 Images) | 56.7 seconds |
| Context Length | 16,384 tokens |
| KV Cache (K) | q8_0 |
| GPU | NVIDIA RTX PRO 6000 Blackwell Server Edition |
| GPU VRAM Capacity | 96 GB |
| Peak GPU VRAM Usage | 27,661 MiB (27.01 GiB) |
| Peak GPU Utilization | 96% |
| GPU Layers Offloaded | 99 (-ngl 99) |
| Random Seed | 42 |
| Speculative Decoding | MTP (draft_n_max=2) |
| Vision Projector | Qwen3.6-27B-abliterated-NVFP4.mmproj.gguf |
| Test Images | 3 |
Test Shots
| Shot | Task | Prompt Processing (tokens/s) | Generation (tokens/s) | Wall Time |
|---|---|---|---|---|
| 1 | Vision OCR - Image 1 | 1,042.72 | 98.58 | 15.42 s |
| 2 | Vision OCR - Image 2 | 1,181.32 | 102.68 | 19.57 s |
| 3 | Vision OCR - Image 3 | 1,249.76 | 101.97 | 21.70 s |
Run Inference
# Download a test image
wget -O /path/to/models/input.jpeg \
"https://cdn-uploads.huggingface.co/production/uploads/65bb837dbfb878f46c77de4c/EF8cldKHhUEO3MZPZSD19.jpeg"
# Run Qwen3.6-27B Abliterated NVFP4 with vision support
/data/llama.cpp/build/bin/llama-cli \
-m /path/to/models/Qwen3.6-27B-abliterated-NVFP4.gguf \
--mmproj /path/to/models/Qwen3.6-27B-abliterated-NVFP4.mmproj.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
-ngl 99 \
-c 16384 \
--cache-type-k q8_0 \
--seed 42 \
-n 8192 \
-st \
--image /path/to/models/input.jpeg \
-p "OCR the image."
Performance results reflect this specific hardware and inference configuration. Actual throughput may vary depending on the input image, prompt, model, software build, and runtime settings.
Snapshots (3)
License and Attributions
Base Model: Qwen/Qwen3.6-27B.
Source Model: huihui-ai/Huihui-Qwen3.6-27B-abliterated: Used for NVFP4 compression and quantization.
Quantization: Performed using
llama.cppby prithivMLmods.License: This model follows the Apache 2.0 license of the original Qwen/Qwen3.6-27B base model.
llama.cpp
LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp