tags:
- text-generation-inference
- llama-cpp
- abliterated
- nvfp4
- nvidia
- llama-cli
- nvidia-blackwell
- rtx-pro-6000
- SM120
- GB202
- B200
- B300
- FP4
- blackwell
- qwen3_5
- Qwen3.5
- Qwen3.6
- Qwen3.8
base_model: - Qwen/Qwen3.5-4B
pipeline_tag: text-generation
license: apache-2.0
language: - en
library_name: transformers
Qwen3.5-4B-abliterated-NVFP4-GGUF
Qwen3.5-4B from Alibaba's Qwen team is a compact 4B-parameter dense multimodal reasoning model with a native 262K context window (extensible to 1M+ tokens), featuring a hybrid Gated DeltaNet + Gated Attention architecture, 248K vocabulary for 201 languages, multi-token prediction, and early-fusion training for unified text, image, and video understanding. Despite its small size, it achieves remarkable intelligence—27 on the Artificial Analysis Intelligence Index, outperforming all models under 5B parameters including Falcon-H1R-7B and NM-Nano-9B-V2—while delivering 65.4% on MMMU-Pro multimodal reasoning, 77 on the Intelligence Index, and comparable performance to GPT-OSS-20B on complex reasoning at roughly 5× fewer parameters. Apache 2.0-licensed and optimized for consumer hardware (~3GB VRAM in 4-bit quantization), it excels at agentic workflows with native tool calling, OCR, document parsing, visual question answering, and code generation, supporting vLLM/Ollama/llama.cpp for edge-to-server deployment as the most capable sub-5B multimodal model available.
[!NOTE]
MTP — Speculative Decoding: No / Text Only / Accelerated FP4 Tensor Core Performance
Model Files
| File Name | Quant Type | File Size | File Link | Description |
|---|---|---|---|---|
| Qwen3.5-4B-abliterated-NVFP4.gguf | NVFP4 (Abliterated) | 4.75 GB | Link | NVFP4 quantization of the abliterated (uncensored/guardrail-removed) version. Highly optimized for NVIDIA Blackwell hardware accelerators (e.g., RTX Pro 6000, RTX 50-series, B200, B300), offering native hardware-accelerated FP4 Tensor Core speeds. |
NVFP4 Test Run
Tested Qwen3.5-4B-abliterated-NVFP4.gguf on an NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with 96 GB of VRAM, using llama-cli with full GPU offloading.
Performance Metrics
| Metric | Value |
|---|---|
| Average Prompt Processing Speed | 332.69 tokens/s |
| Average Generation Speed | 196.16 tokens/s |
| Total Wall Time | 38.4 seconds |
| Context Length | 8,192 tokens |
| KV Cache (K) | q8_0 |
| GPU | NVIDIA RTX PRO 6000 Blackwell Server Edition |
| GPU VRAM Capacity | 96 GB |
| Peak GPU VRAM Usage | 5.50 GiB |
| Peak GPU Utilization | 89% |
| GPU Layers Offloaded | 99 (-ngl 99) |
| Random Seed | 42 |
| Speculative Decoding | No |
| Test Runs | 5 |
Test Shots
| Shot | Test |
|---|---|
| Shot 1 | C: Insert Node at End of Singly Linked List |
| Shot 2 | Jogger and Train Passing Time |
| Shot 3 | Two Pipes Filling a Tank |
| Shot 4 | Students in Examination Rooms |
| Shot 5 | Father and Son Ages |
Run Inference
./build/bin/llama-cli \
-m /path/to/models/Qwen3.5-4B-abliterated-NVFP4.gguf \
-ngl 99 \
-c 8192 \
--cache-type-k q8_0 \
--seed 42 \
-n 8192 \
-st \
-p "Your prompt here"
Performance results reflect this specific hardware and inference configuration. Actual throughput may vary depending on the prompt, model, software build, and runtime settings.
Snapshots (5)
llama.cpp
LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp
Quantized using llama.cpp by prithivMLmods.