license: apache-2.0
base_model: Bahushruth/Qwen3.6-35B-A3B-abliterated-v4
tags:
- abliteration
- uncensored
- qwen3
- moe
- gguf
- llama-cpp
- ollama
model_type: qwen3_moe
pipeline_tag: text-generation
datasets: - Bahushruth/abliteration-harmful-enriched
Qwen3.6-35B-A3B-abliterated-v4-GGUF
GGUF quantizations of Bahushruth/Qwen3.6-35B-A3B-abliterated-v4 for llama.cpp, Ollama, LM Studio, KoboldCPP, and other GGUF-compatible runtimes.
Uncensored model — refusal behavior removed via norm-preserving abliteration (0% refusal, full capability preservation). See the base model card for method details.
Blog post: Abliteration: Uncensoring LLMs via Weight Surgery
Quantizations
All standard quants are compatible with Ollama, LM Studio, KoboldCPP, and llama.cpp out of the box — no special flags needed.
| File | Size | RAM Required | Notes |
|---|---|---|---|
...-BF16.gguf |
69 GB | 80+ GB | Full precision |
...-Q8_0.gguf |
37 GB | 48+ GB | Near-lossless |
...-Q6_K.gguf |
29 GB | 40+ GB | Very high quality |
...-Q5_K_M.gguf |
25 GB | 32+ GB | Recommended for 48GB systems |
...-Q4_K_M.gguf |
21 GB | 24+ GB | Good quality, fits 24GB |
...-IQ4_XS.gguf |
19 GB | 24+ GB | High quality 4-bit (imatrix) |
...-IQ4_NL.gguf |
20 GB | 24+ GB | Non-linear 4-bit (imatrix) |
...-Q3_K_M.gguf |
17 GB | 20+ GB | Good for 16GB VRAM GPUs |
...-IQ3_M.gguf |
15 GB | 20+ GB | High quality 3-bit (imatrix) |
...-IQ3_XXS.gguf |
14 GB | 16+ GB | Smallest usable 3-bit (imatrix) |
...-Q2_K.gguf |
13 GB | 16+ GB | 2-bit, quality tradeoffs |
...-IQ2_M.gguf |
12 GB | 16+ GB | Smallest, significant quality loss |
MTP (Multi-Token Prediction)
Qwen3.6-35B-A3B includes an MTP draft head (blk.40) for speculative decoding. All standard quants above exclude MTP for maximum compatibility.
A separate BF16-MTP file is available for advanced users running llama-server directly:
| File | Size | Notes |
|---|---|---|
...-BF16-MTP.gguf |
~71 GB | BF16 with MTP draft head included |
To use MTP speculative decoding:
./llama-server -m Qwen3.6-35B-A3B-abliterated-v4-BF16-MTP.gguf \
--jinja --spec-type draft-mtp --spec-draft-n-max 1 -ngl 99
Runtime compatibility: MTP requires llama-server b9180+. Ollama does not support MTP yet. KoboldCPP 1.116.1+ and current LM Studio handle MTP fine, but many users are on older installs — the standard no-MTP quants are the safe default.
Quantization Types Explained
K-quants use a block-wise quantization scheme where weights are grouped into blocks and each block gets its own scale factor, preserving more precision than naive round-to-nearest quantization.
| Suffix | Meaning |
|---|---|
Q8_0 |
8-bit uniform quantization (simplest, largest) |
Q6_K |
6-bit k-quant. All tensor blocks use 6-bit. |
Q5_K_M |
5-bit k-quant, medium variant. Important tensors bumped to Q6_K. |
Q4_K_M |
4-bit k-quant, medium variant. Important tensors kept at Q5_K. |
Q3_K_M |
3-bit k-quant, medium variant. Important tensors at Q4_K. |
Q2_K |
2-bit k-quant. Aggressive compression with quality tradeoffs. |
IQ4_XS |
4-bit importance-matrix quant. Uses imatrix calibration for better quality at same size. |
IQ4_NL |
4-bit non-linear quant. Non-uniform quantization levels tuned to weight distributions. |
IQ3_M |
3-bit importance-matrix quant. Medium quality. |
IQ3_XXS |
3-bit importance-matrix quant. Extra small — minimum viable 3-bit. |
IQ2_M |
2-bit importance-matrix quant. Extreme compression. |
The _M (medium) suffix means a mixed-precision strategy: less important layers get the headline bit-width while critical layers (attention output, embeddings) get one level higher. This gives noticeably better quality than pure _S (small) variants at only ~5-10% size increase.
IQ quants use importance-matrix calibration (computed from wikitext-2) to allocate more precision to weights that matter most for model output quality. They achieve better perplexity than K-quants at the same file size.
Quickstart — Ollama
# Recommended for Apple Silicon 48GB+ (M4 Pro, M4 Max, etc.)
ollama run hf.co/Bahushruth/Qwen3.6-35B-A3B-abliterated-v4-GGUF:Q5_K_M
# For 24GB systems
ollama run hf.co/Bahushruth/Qwen3.6-35B-A3B-abliterated-v4-GGUF:Q4_K_M
# For 10GB VRAM (RTX 3080)
ollama run hf.co/Bahushruth/Qwen3.6-35B-A3B-abliterated-v4-GGUF:IQ3_XXS
Usage — llama.cpp
huggingface-cli download Bahushruth/Qwen3.6-35B-A3B-abliterated-v4-GGUF \
Qwen3.6-35B-A3B-abliterated-v4-Q5_K_M.gguf --local-dir .
# Basic usage
./llama-cli -m Qwen3.6-35B-A3B-abliterated-v4-Q5_K_M.gguf \
-p "You are a helpful assistant." \
--chat-template chatml -cnv
Important Notes
- This is a MoE model (256 experts, 8 active per token). Despite 35B total params, only ~3B are active — efficient for its capability level.
- All standard quants work with Ollama, LM Studio, KoboldCPP, and llama.cpp without any special flags.
- IQ quants (IQ4_XS, IQ3_M, etc.) use imatrix calibration for better quality-per-bit.
- Tested and confirmed working on M4 Pro 48GB with Q5_K_M, and RTX 3080 10GB with IQ3_XXS.
Conversion Details
- Converter:
llama.cpp/convert_hf_to_gguf.py --no-mtp - Quantizer:
llama-quantize - imatrix: wikitext-2, 100 chunks, 16 threads
- Infrastructure: Modal (CPU-only, 16 cores, 128GB RAM)
Disclaimer
This model has had safety guardrails removed. Released for research purposes. The creator assumes no responsibility for downstream use.