language:
- en
- it
license: apache-2.0
base_model: - Qwen/Qwen3.6-35B-A3B
- ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ
- huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated
tags: - gguf
- llama.cpp
- moe
- abliterated
- uncensored
- quantization
- 2-bit
library_name: gguf
pipeline_tag: text-generation
arxiv: 2604.18556
Qwen3.6-35B-A3B — GSQ 2-bit experts + abliterated trunk
Hybrid 35B-A3B MoE: the 2-bit GSQ expert weights kept bit-exact, wrapped in huihui's abliterated trunk,
with the expert down-projections abliterated per layer. 11.58 GB, 2.67 bits per weight, vanilla llama.cpp.
Experimental model. It started as an attempt to build a Qwen3.6-35B-A3B from what I consider the best
2-bit weights available — ISTA-DASLab's GSQ quantization
(ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ) —
but in abliterated form, taking the abliterated bits from huihui-ai
(huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated).
The result is a hybrid: the GSQ experts kept at 2 bits, huihui's abliterated trunk, and a per-layer
abliteration of the expert down-projections.
Unofficial derivative — not affiliated with Alibaba/Qwen, ISTA-DASLab or huihui-ai.
Uncensored model. Use responsibly and legally.
At a glance
| File | 11.59 GB · 34.66B params · 2.672 bpw |
| Experts gate/up | 2.25 bpw, Q2_0 — lossless transcode of the GSQ values |
| Experts down | 2.62 bpw, Q2_K |
| Trunk | Q5_K, imatrix-guided |
| Embeddings / output | Q8_0 / Q6_K |
| Router | F32 |
| Context | 262,144 |
| Speed, M3 Pro | 42-45 tok/s decode · 225 tok/s prefill |
| Perplexity (wikitext-2, 40×512) | 6.9247 ±0.16 |
| MTP block | pruned |
Weight distribution
| Tensors | Count | Type | bpw | Size | % of file |
|---|---|---|---|---|---|
ffn_gate_exps |
40 | Q2_0 |
2.25 | 3.020 GB | 26.1% |
ffn_up_exps |
40 | Q2_0 |
2.25 | 3.020 GB | 26.1% |
ffn_down_exps |
40 | Q2_K |
2.62 | 3.523 GB | 30.4% |
token_embd |
1 | Q8_0 |
8.50 | 0.540 GB | 4.7% |
output |
1 | Q6_K |
6.56 | 0.417 GB | 3.6% |
attn_qkv |
30 | Q5_K |
5.50 | 0.346 GB | 3.0% |
attn_gate, ssm_out |
30 + 30 | Q5_K |
5.50 | 0.173 + 0.173 GB | 3.0% |
attn_q, attn_output |
10 + 10 | Q5_K |
5.50 | 0.115 + 0.058 GB | 1.5% |
ffn_gate_inp (router) |
40 | F32 |
32.0 | 0.084 GB | 0.7% |
ffn_{gate,up,down}_shexp |
40 each | Q5_K |
5.50 | 0.029 GB each | 0.6% |
norms, attn_k/v, ssm_alpha/beta |
51 | F32 / Q5_K |
— | 0.060 GB | 0.5% |
By provenance:
| Source | Parameters | Bytes |
|---|---|---|
| ISTA-DASLab GSQ 2-bit — exact (gate/up) | 62.0% | 52.1% |
| ISTA-DASLab GSQ 2-bit — abliterated, re-encoded (down) | 31.0% | 30.4% |
| huihui-ai abliterated (trunk, embd, output, router) | 7.1% | 17.4% |
Build steps
| Step | What | Result |
|---|---|---|
| 1 | GSQ gate/up experts → Q2_0, verified block-by-block against source |
byte-identical packing, 2.25 bpw |
| 2 | Expert down_proj abliteration: rank-1, per layer, direction from donor Gram |
39/40 layers, ‖ΔW‖/‖W‖ = 1.7%, row norms preserved |
| 3 | Abliterated down → Q2_K |
2.62 bpw |
| 4 | Trunk quantized with imatrix (510 tensors, 76 chunks) | Q5_K attn/SSM/shared, Q6_K output, Q8_0 embd, F32 router |
| 5 | MTP block pruned after the per-layer KV fix (block count, recurrent array) | 733 tensors, block_count 40 |
Files
| File | Size | SHA-256 |
|---|---|---|
Qwen3.6-35B-A3B-GSQ-2bit-abliterated-2.672bpw.gguf |
11.59 GB | b10ae67d02f4e80ba0cef862006925814cb436e6eb02407dd26b257bdb59831c |
Results
Perplexity, wikitext-2, 40 chunks × 512, one harness:
| Build | Size | PPL |
|---|---|---|
| This file | 11.59 GB | 6.9247 |
| Same pipeline, Q8_0 trunk | 13.46 GB | 6.9293 |
| Same source, gate/up 4.5 bpw + down 5.56 bpw | 21.71 GB | 6.9906 |
| Reference: public Q3_K from true BF16 weights | 17.17 GB | 6.1898 |
The source is already 2-bit: extra precision preserves it, it does not add information. The last row is what
true BF16 weights buy (≈0.73 PPL) — the cost of the 2-bit source, not of this file.
Battery (single sample, thinking off): probe correct, Italian prose coherent, code runs, no repetition,
compliant on a safety probe the base model refuses.
Limits
- Down-projections carry the only meaningful re-quantization error (no 4-level grid after abliteration).
- Abliteration is spot-checked; no refusal benchmark; thinking-off only. Thinking is ON by default upstream.
- PPL: one corpus, one context length, one harness — relative, not absolute.
- Text-only. No vision tower. MTP pruned.
Requirements
- GGUF runtime with
Q2_0(ggml type 42) — llama.cpp builds from late 2026. - No patched build needed to run it. Do not re-quantize with stock
llama-quantize. llama-server -m <file> -c 16384 --flash-attn -ngl 99
License
Apache-2.0, with attribution to Qwen (architecture, tokenizer), ISTA-DASLab (GSQ 2-bit experts) and
huihui-ai (abliterated trunk/embeddings). No endorsement implied.
Credits
- Build, expert abliteration, quantization, measurements — Ermzzz999.
- Base model — Qwen Team, Qwen/Qwen3.6-35B-A3B
(architecture and tokenizer). - 2-bit expert weights — ISTA-DASLab,
ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ. - Abliterated trunk and embeddings — huihui-ai,
huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated. - Runtime and quantizer — llama.cpp / ggml (the
Q2_0type
used here, and the K-quant kernels).
If you use this model, please cite the quantization method — paper:
arXiv:2604.18556.
@article{gsq2026,
title = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and
Kurti{\'c}, Eldar and Kleinegger, Max and Alistarh, Dan},
journal = {arXiv preprint arXiv:2604.18556},
year = {2026},
url = {https://arxiv.org/abs/2604.18556}
}