license: apache-2.0
base_model:
- prism-ml/Ternary-Bonsai-2-27B-gguf
- BoldingBuilds/Ternary-Bonsai-2-27B-Abliterated-PTQ1_0-GGUF
tags: - gguf
- prismml
- pq2_0
- ternary
- abliterated
pipeline_tag: text-generation
Ternary-Bonsai-2-27B Abliterated — PQ2_0
A PQ2_0 build of the refusal-ablated Ternary Bonsai 2 27B. It exists because the
existing abliterated release is PTQ1_0, and PTQ1_0 has GPU kernels for NVIDIA Ampere
only — on every other backend it falls back to fp16 dequantise + BLAS.
On an AMD MI210 (gfx90a / CDNA2) that difference is 2.2x:
| build | decode |
|---|---|
| Abliterated PTQ1_0 (existing release) | 23.8 tok/s |
| Abliterated PTQ1_0 + MMVQ enabled for HIP (local patch) | 23.8 tok/s |
| PTQ1_0, stock, before patch | 18.2 tok/s |
| This build (PQ2_0) | 51.2 tok/s |
Measured single-stream, -ngl 99 -c 8192, one MI210, llama.cpp PrismML fork.
Why PTQ1_0 is slow off Ampere
In the PrismML fork, MMQ tile configs exist per architecture. PTQ1_0 has entries for Ampere
and nothing else:
mmq-config-ampere.cuh PTQ1_0=11 PQ2_0=16 Q2_0=16
mmq-config-cdna.cuh PTQ1_0=0 PQ2_0=7 Q2_0=7
mmq-config-rdna2/3/4 PTQ1_0=0 PQ2_0=12 Q2_0=12
mmq-config-pascal.cuh PTQ1_0=0 PQ2_0=11 Q2_0=11
PTQ1_0 is additionally #if !defined(GGML_USE_HIP)-gated in 17 places, and its eligibility
check is turing_mma_available(cc), which is hard-wired to NVIDIA. PQ2_0 has CDNA kernels
and hits the fast path.
How this was made
The abliterated weights only exist as PTQ1_0, so the edit was recovered rather than
re-derived:
- Dequantised both the abliterated and stock PTQ1_0 checkpoints.
- Diffed them. Exactly 98 tensors differ — writers only: 49
ffn_down, 36ssm_out,
13attn_output. This matches the original release's stated scope exactly. - Took the top singular vector of each delta. All 98 share one direction
(|cos(d_i, d_0)|mean 0.946; agreement with the consensus direction mean 0.988,
min 0.955) — consistent with the stated rank-1 edit. - Applied
W <- W - d (dᵀ W)to the F16 checkpoint on those same 98 tensors. - Requantised F16 -> PQ2_0 with the fork's
llama-quantize.
Working from F16 rather than from the ternary checkpoint avoids compounding quantisation
error: the recovered direction is clean even though the delta it came from is not, because
the per-tensor quantisation noise averages out across 98 tensors.
What is and is not verified
Verified:
- The correct 98 tensors were edited, along a direction validated at 0.99 consensus.
- Capability is intact — 4/4 on a small factual/arithmetic probe, coherent long-form output.
- 51.2 tok/s on one MI210, correctness stable across runs.
NOT verified: the refusal rate. A comparison against the stock PQ2_0 build returned
0/6 refusals for both, so the probe could not discriminate — the stock model already
complies with mild prompts, and measuring this properly needs the kind of prompt set this
build was not evaluated against. Treat the refusal behaviour as unmeasured, not as
equivalent to the PTQ1_0 release it derives from.
Note also that PQ2_0 here is 2.45 bpw rather than the stock file's 2.13: the projection
de-ternarises the weights, so the quantiser can no longer represent them losslessly.
Requirements
Needs the PrismML llama.cpp fork (prismml-eng/llama.cpp, branch prism). Stock
llama.cpp cannot load PQ2_0. Build for CDNA2:
cmake -S . -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx90a \
-DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF
cmake --build build -j
Attribution
Created using Bonsai by Prism ML. Built from Qwen3.8-27B, Copyright 2026 Alibaba Cloud
(Apache 2.0). The ablation edit derives fromBoldingBuilds/Ternary-Bonsai-2-27B-Abliterated-PTQ1_0-GGUF (Apache 2.0).
Apache 2.0. See LICENSE and NOTICE.txt.