license: apache-2.0
base_model: Hikari07jp/Ternary-Bonsai-27B-Abliterated-LowDeg-GGUF
library_name: llama-cpp
pipeline_tag: text-generation
tags:

• qwen
• llama.cpp
• ternary
• abliterated
• uncensored
• bonsai
• prism

Ternary-Bonsai-27B — Abliterated (LowDeg), in the native 2-bit packing

PrismML's Ternary-Bonsai-27B stores every weight as −1, 0 or
+1. The abliteration — the refusals removed by flipping a fraction of those digits — is
Hikari07jp's work, and this repository
re-publishes that build with a different packing.

What is mine here is the packing, not the ablation: the three files below differ in how much of the feed-forward
network is left in 4 bits, and the two patched ones are half the size of the 4-bit build with a better perplexity.
They also carry a fix that makes the file loadable at all (see below).

tensor_types_p0123.txt records, tensor by tensor, the packing the patched variant ends up with: the ffn_gate and
ffn_up of blocks 0 to 3 stay q4_0, the 482 other weight tensors are the native 2-bit ternary codec
(Q2_0, group 128, 2,125 bits per weight), and the norms stay in fp32.

Credit, precisely: base model prism-ml/Ternary-Bonsai-27B-gguf
(Prism ML) → ablation Hikari07jp/Ternary-Bonsai-27B-Abliterated-LowDeg-GGUF
(Hikari07jp) → the packings in this repository, and the retag fix (me).

Two things the file names do not tell you

1. These are not 1-bit files. general.file_type says 41 (Q1_0), but no tensor carries that id: the weights
are the 2-bit ternary codec. A ternary weight needs its sign and whether it is zero, so 2 bits per weight is the
floor for a lossless packing of this model — the "Q1_0" in the name is only in the metadata. The measured size
confirms it: 2,125 bits per weight, 6,87 GiB for a 27B.

That is also why the patched files are better than the 4-bit one, not merely smaller: a 4-bit re-quantisation of a
ternary model throws away the fact that the fourth bit never carried anything. Same weights, same protocol:

The perplexity does not depend on where the layers run: on p0to7, 100 % on the GPU, 26 layers on the GPU, or a single
layer on the GPU give 10,3288 / 10,3288 / 10,3287.

2. The published files load as they are; the earlier revisions did not. The ternary weights are stored under the
Prism type id 142 (PQ2_0, group 128), which the current fork reads natively: the three files published here are
downloaded and run without any conversion — llama-server --model <file> is all it takes (the 4-bit build still asks
for a 16 GB card, see the table above; the two patched ones run on 8 GB).

The earlier revisions of this repository used the legacy Prism type id 42 (Q2_0, group 128) instead, while
the current fork reads type 42 as the official group-64 Q2_0, and refused the file:

E gguf_init_from_reader: this file matches the legacy Prism Q2_0 layout (group size 128 stored as ggml type id 42),
 but this build reads Q2_0 as the official group-64 format
E gguf_init_from_reader: you are probably using the wrong GGUF: use the PQ2_0 version of this model (ggml type id 142)

The fix is a tag, not a conversion: the 2-bit codec is the same, only the type id differs (42 → 142, four bytes per
tensor, header only). tools/retag42.py does it and prints the sha256 of the data section before and after, so you
can check that the weights are the published ones:

$ python3 tools/retag42.py Ternary-Bonsai-27B-Abliterated-LowDeg-Q4_0-p0123.gguf
  490 tensors retagged 42 -> 142
  data section sha256 before : 0080b09b…
  data section sha256 after  : 0080b09b…   (identical: the weights did not move)

--revert puts the legacy tag back. The retag is only needed for the older revisions — the ones published before
commit 4e9e0fa — and it leaves the data section untouched, so the weights measured below are the very same ones
(the sha256 of the data section is identical before and after).

Measured

Perplexity: wikitext-2-raw, 18 944 tokens, 37 chunks of 512, llama-perplexity -m M -f ppl_corpus_wikitext2_78k.txt -c 512 -fit on,
the same protocol for every file, the same corpus for all three. Decode rate: 200 generated tokens per run, two runs
per file, counted with the server's own tokenizer and cross-checked against the server's internal counter. Card:
RTX 3070 8 GB, desktop running.

At 32 768 the engine refuses: the recurrent-state buffer (149,6 MiB) does not fit.

Running it

The Bonsai Hadamard packing needs a llama.cpp that supports it; upstream llama.cpp does not read these files
(invalid ggml type). PrismML's fork does.

llama-server \
  --model Ternary-Bonsai-27B-Abliterated-LowDeg-Q4_0-p0123.gguf --alias bonsai27-abliterated \
  --ctx-size 16384 --cache-type-k q4_0 --cache-type-v q4_0 \
  --n-gpu-layers 65 --fit off --flash-attn on --jinja -b 512 -ub 128 --parallel 1

• --n-gpu-layers 65, not 64: the count includes the output layer, and 64 leaves the first block on the CPU,
  which switches the fused GDN off.
• Alias it. The file's internal name is Bonsai-27B, so without an alias it is indistinguishable from the
  stock build in an API's model list. The registry's convention is the same: an uncensored variant keeps the same
  model entry and is served under its own name.
• The 4-bit build does not fit in 8 GB at all.

License and attribution

Apache-2.0, like the base model. This is a derivative of
prism-ml/Ternary-Bonsai-27B-gguf (Apache-2.0, itself
derived from Qwen); the base repository's LICENSE and NOTICE.txt apply unchanged and are reproduced here.
Prism ML asks for the attribution "Created using Bonsai by Prism ML" when the software is publicly deployed or
redistributed — this repository is such a redistribution.