library_name: transformers
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3.5-9B/blob/main/LICENSE
pipeline_tag: text-generation
base_model:
- huihui-ai/Huihui-Qwen3.5-9B-abliterated
tags: - abliterated
- uncensored
- exl3
- exllamav3
- qwen3.5
- gated-delta-net
- hybrid-attention
- tabbyapi
quantized_by: abcdsystems
Qwen3.5-9B-Abliterated-GDN-Hybrid — EXL3 6.0bpw
EXL3 (exllamav3) quantization of huihui-ai/Huihui-Qwen3.5-9B-abliterated at 6.0 bits per weight.
Quantized by ABC&D System Inc for production real-time inference on a single RTX 3090.
Model Details
| Property | Value |
|---|---|
| Base model | Qwen/Qwen3.5-9B |
| Abliteration | huihui-ai via remove-refusals-with-transformers |
| Parameters | 9B dense (all active per token) |
| Architecture | Qwen3_5ForConditionalGeneration — GatedDeltaNet + Attention hybrid |
| Quantization | EXL3 6.0bpw, head bits 6, codebook mul1 |
| Calibration | 250 rows × 2048 cols |
| VRAM loaded | ~5.6GB model + KV cache |
| Total size on disk | ~8.5GB |
| License | Apache 2.0 (Alibaba Cloud) |
Architecture — Why This Model Is Different
Qwen3.5 is not a standard transformer. It uses a hybrid attention design that alternates between two fundamentally different attention mechanisms:
GatedDeltaNet Linear Attention (layers 0, 1, 2, 4, 5, 6, 8, 9, 10, ...)
- O(n) complexity instead of O(n²) — scales linearly with sequence length
- Recurrent state via conv1d kernel (kernel dim 4)
- 16 key heads × 128 dim, 32 value heads × 128 dim
- Gated with learned input/output projections + beta decay
Full Quadratic Attention (layers 3, 7, 11, 15, 19, 23, 27, 31)
- Standard grouped-query attention (16 heads, 4 KV heads, head dim 256)
- RoPE with partial rotary factor 0.25, theta 10M
- Provides global context every 4th layer
Layer Pattern
[linear, linear, linear, FULL, linear, linear, linear, FULL, ...] × 4 = 32 layers
24 GatedDeltaNet layers + 8 full attention layers. The linear layers handle local/sequential patterns efficiently while the full attention layers provide global context anchoring.
Multi-Token Prediction (MTP)
The model includes a 1-layer MTP head for predicting multiple next tokens simultaneously, enabling faster inference throughput than standard single-token autoregressive decoding.
Text-Only Weights
Despite the architecture string Qwen3_5ForConditionalGeneration (which Alibaba uses for all Qwen3.5 variants including VL), this model contains zero vision weights. No ViT encoder, no image projector. The architecture class name reflects the MTP head and hybrid design, not multimodal capability.
Quantization Details
Quantized with exllamav3 v1.1.0.
- Format: EXL3 with trellis-based weight encoding
- Bits per weight: 6.0 (all linear layers uniform)
- Head bits: 6
- Codebook: mul1
- Why EXL3 over GGUF: Native tensor core utilization on Ampere GPUs (RTX 3090/4090). GGUF dequantizes to FP16 at runtime, wasting tensor cores. EXL3 operates natively, yielding ~1.5x faster inference at equivalent quality.
Recommended Serving
TabbyAPI (recommended)
# config.yml
network:
host: 0.0.0.0
port: 11435
model:
model_dir: /path/to/models
model_name: Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw
max_seq_len: 4096
gpu_split: [24]
cache_mode: FP16
CUDA_VISIBLE_DEVICES=0 python3 main.py --config config.yml
Important: Disable Thinking Mode
Qwen3.5 has a built-in chain-of-thought reasoning mode that is on by default. For real-time applications (voice, chat, API), you must disable it or the model will consume your entire token budget on internal reasoning before producing visible output.
Add to your system prompt:
/no_think
Or set enable_thinking: false in your chat template parameters.
Note on exllamav3 Raw API
The raw exllamav3 Generator Python API may fail to initialize GatedDeltaNet recurrent state (conv_state on meta device error). TabbyAPI handles this initialization correctly. Use TabbyAPI as the serving layer rather than calling exllamav3 directly.
Hardware Requirements
| Config | VRAM | Notes |
|---|---|---|
| Minimum | ~8GB | Model only, minimal KV cache |
| Recommended | 16GB+ | Model + FP16 KV cache for 4096 context |
| Tested on | RTX 3090 24GB | 5.6GB model, 18.4GB free for cache |
Performance (tested on RTX 3090)
- Response latency: 300–500ms per completion
- Suitable for real-time voice pipelines with STT overhead
- First request ~14ms (cache warmup, empty response)
Use Cases
This quantization was built for low-latency, single-user dedicated inference — specifically real-time voice AI over SIP/phone calls. The abliteration ensures zero self-censorship in commercial conversation contexts.
Good for:
- Real-time voice assistants and phone agents
- Single-GPU dedicated inference servers
- Applications requiring uncensored, natural conversational output
- Latency-sensitive pipelines (sub-600ms budget)
Not ideal for:
- Multi-user batched serving (use AWQ + vLLM instead)
- Vision/multimodal tasks (no vision weights)
- Applications requiring safety guardrails (abliterated model)
Credits
- Base model: Qwen/Qwen3.5-9B by Alibaba Cloud (Apache 2.0)
- Abliteration: huihui-ai via remove-refusals-with-transformers
- EXL3 quantization: ABC&D System Inc using exllamav3 by turboderp
- Serving: TabbyAPI by theroyallab
Usage Warnings
This model has had its safety filtering surgically removed (abliterated). It will comply with any instruction without refusal. Users are solely responsible for ensuring their use complies with applicable laws and ethical standards. See huihui-ai's original warnings for full details.