tags:
- text-generation-inference
- mtp
- llama-cpp
- abliterated
- qwen3_5_moe
- nvfp4
- nvidia
- llama-cli
- speculative-decoding
- nvidia-blackwell
- rtx-pro-6000
- SM120
- GB202
- B200
- B300
- FP4
- blackwell
- qwen3_5
- Qwen3.5
- Qwen3.6
- Qwen3.8
- 35B
- A3B
base_model: - Qwen/Qwen3.6-35B-A3B
library_name: transformers
license: apache-2.0
language: - en
pipeline_tag: image-text-to-text
Qwen3.6-35B-A3B-abliterated-NVFP4-GGUF
Qwen3.6-35B-A3B from Alibaba's Qwen team is a sparse Mixture-of-Experts (MoE) multimodal language model with 35B total parameters but only ~3B active per token, featuring a hybrid architecture of 40 layers with Gated DeltaNet blocks, 2048 hidden dimension, 248K vocabulary for 201+ languages, multi-token prediction (MTP), and 262K native context length (extensible to 1M+ tokens via YaRN). Released as a coding/agentic powerhouse, it achieves SOTA agentic coding performance—73.4% on SWE-Bench Verified (rivaling models 10x its active size), frontier-level repository reasoning, frontend workflows, and multimodal understanding (text/images/video/OCR)—while delivering dense-3B inference costs with FP8 quants for single-GPU deployment via vLLM/Ollama. Apache 2.0-licensed with optimizations like prefix caching, chunked prefill, and tool-call parsing, the upgraded Qwen3.6 iteration over Qwen3.5-35B-A3B prioritizes production reliability for autonomous dev agents, structured JSON outputs, and long-context software engineering tasks.
[!NOTE]
MTP — Speculative Decoding: Yes
[!NOTE]
Multi-Token Prediction (MTP) GGUF is a specialized GGUF model file format extension that integrates speculative decoding directly into the model weights to significantly accelerate local inference. Unlike traditional speculative decoding which requires a separate, smaller "draft" model, MTP GGUF files include additional output heads within the main model architecture that predict multiple future tokens in a single forward pass.
Model Files
| File Name | Quant Type | File Size | File Link | Description |
|---|---|---|---|---|
| Qwen3.6-35B-A3B-abliterated-NVFP4.gguf | NVFP4 | 22.7 GB | Link | NVFP4 quantization. Highly optimized for NVIDIA Blackwell hardware accelerators (e.g., RTX Pro 6000, RTX 50-series, B200, B300), offering native hardware-accelerated FP4 Tensor Core speeds and a strong balance of size and performance for this 35B parameter model (A3B architecture). |
| Qwen3.6-35B-A3B-abliterated-NVFP4.mmproj.gguf | mmproj | 903 MB | Link | Multimodal projection file. Used for vision/language models. |
NVFP4 Test Run
Tested Qwen3.6-35B-A3B-abliterated-NVFP4.gguf on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB of VRAM, using llama-cli with full GPU offloading and MTP speculative decoding.
Performance Metrics
| Metric | Value |
|---|---|
| Average Prompt Processing Speed | 164.58 tokens/s |
| Average Generation Speed | 272.99 tokens/s |
| Total Wall Time | 97.2 seconds |
| Context Length | 16,384 tokens |
| KV Cache (K) | q8_0 |
| GPU | NVIDIA RTX PRO 6000 Blackwell Server Edition |
| GPU VRAM Capacity | 96 GB |
| Peak GPU VRAM Usage | 21.73 GiB |
| Peak GPU Utilization | 82% |
| GPU Layers Offloaded | 99 (-ngl 99) |
| Random Seed | 42 |
| Speculative Decoding | MTP (draft_n_max=2) |
| Test Runs | 5 |
Test Shots
| Shot | Test |
|---|---|
| Shot 1 | C: Insert Node at End of Singly Linked List |
| Shot 2 | Jogger and Train Passing Time |
| Shot 3 | Two Pipes Filling a Tank |
| Shot 4 | Students in Examination Rooms |
| Shot 5 | Father and Son Ages |
Run Inference
./build/bin/llama-cli \
-m /path/to/models/Qwen3.6-35B-A3B-abliterated-NVFP4.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
-ngl 99 \
-c 16384 \
--cache-type-k q8_0 \
--seed 42 \
-n 8192 \
-st \
-p "Your prompt here"
Performance results reflect this specific hardware and inference configuration. Actual throughput may vary depending on the prompt, model, software build, and runtime settings.
llama.cpp
LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp
Snapshots (5)
License and Attributions
Base Model: Qwen/Qwen3.6-35B-A3B.
Source Model: huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated: Used for NVFP4 compression and quantization.
Quantization: Performed using
llama.cppby prithivMLmods.License: This model follows the Apache 2.0 license of the original Qwen/Qwen3.6-35B-A3B base model.