tags:
- text-generation-inference
- llama-cpp
- abliterated
- nvfp4
- nvidia
- llama-cli
- nvidia-blackwell
- rtx-pro-6000
- SM120
- GB202
- B200
- B300
- FP4
- blackwell
- qwen3_5
- Qwen3.5
- Qwen3.6
- Qwen3.8
pipeline_tag: text-generation
library_name: transformers
license: apache-2.0
language: - en
base_model: - Qwen/Qwen3.5-9B
Qwen3.5-9B-abliterated-NVFP4-GGUF
Qwen3.5-9B from Alibaba's Qwen team is a 9B-parameter dense multimodal language model featuring a hybrid Gated DeltaNet + Gated Attention architecture with 262K native context window (extensible to 1M+ tokens via RoPE scaling), 248K vocabulary supporting 201 languages, and early‑fusion training for unified text, image, and video understanding. It achieves SOTA performance across modalities with 89.2% OCRBench, 84.5% VideoMME, 78.9% MathVision, and 70.1% MMMU-Pro, while delivering production-level agentic capabilities including 66.1% BFCL-V4 and 79.1% TAU2-Bench for native tool calling, plus toggleable thinking mode for step-by-step reasoning on complex tasks. Apache 2.0-licensed and optimized for vLLM/SGLang/llama.cpp/Ollama deployment (~18GB VRAM BF16, ~5GB 4-bit), the instruction-tuned variant excels at repository-level coding, frontend development, document/PDF parsing, visual question answering, and multilingual chatbots as a scalable foundation for edge-to-server multimodal agents.
[!NOTE]
MTP — Speculative Decoding: No / Text Only / Accelerated FP4 Tensor Core Performance
Model Files
| File Name | Quant Type | File Size | File Link | Description |
|---|---|---|---|---|
| Qwen3.5-9B-abliterated-NVFP4.gguf | NVFP4 (Abliterated) | 10.3 GB | Link | NVFP4 quantization of the abliterated (uncensored/guardrail-removed) version. Highly optimized for NVIDIA Blackwell hardware accelerators (e.g., RTX Pro 6000, RTX 50-series, B200, B300), offering native hardware-accelerated FP4 Tensor Core speeds. |
NVFP4 Test Run
Tested Qwen3.5-9B-abliterated-NVFP4.gguf on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB of VRAM, using llama-cli with full GPU offloading.
Performance Metrics
| Metric | Value |
|---|---|
| Average Prompt Processing Speed | 318.37 tokens/s |
| Average Generation Speed | 135.66 tokens/s |
| Total Wall Time | 228.5 seconds |
| Context Length | 8,192 tokens |
| KV Cache (K) | q8_0 |
| GPU | NVIDIA RTX PRO 6000 Blackwell Server Edition |
| GPU VRAM Capacity | 96 GB |
| Peak GPU VRAM Usage | 8.81 GiB |
| Peak GPU Utilization | 93% |
| GPU Layers Offloaded | 99 (-ngl 99) |
| Random Seed | 42 |
| Speculative Decoding | No |
| Test Runs | 5 |
Test Shots
| Shot | Test |
|---|---|
| Shot 1 | C: Insert Node at End of Singly Linked List |
| Shot 2 | Jogger and Train Passing Time |
| Shot 3 | Two Pipes Filling a Tank |
| Shot 4 | Students in Examination Rooms |
| Shot 5 | Father and Son Ages |
Run Inference
./build/bin/llama-cli \
-m /path/to/models/Qwen3.5-9B-abliterated-NVFP4.gguf \
-ngl 99 \
-c 8192 \
--cache-type-k q8_0 \
--seed 42 \
-n 8192 \
-st \
-p "Your prompt here"
Performance results reflect this specific hardware and inference configuration. Actual throughput may vary depending on the prompt, model, software build, and runtime settings.
Snapshots (5)
llama.cpp
LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp
Quantized using llama.cpp by prithivMLmods.