license: apache-2.0
library_name: llama.cpp
pipeline_tag: text-generation
base_model:

• prism-ml/Ternary-Bonsai-2-27B-gguf
  tags:
• ternary
• gguf
• llama-cpp
• abliterated
• uncensored
• bonsai
• hybrid-attention

Ternary-Bonsai-2-27B — abliterated (still pure ternary)

This is an abliterated version of prism-ml/Ternary-Bonsai-2-27B-gguf
(PTQ1_0 packing). It keeps the exact original file size, 5,946,648,928 bytes:

• Every tensor has its original type and shape.
• All 66 edited weight matrices are still {-1, 0, +1} × fp16 group-128 scales.
• On average 0.64% of their trits were flipped, and their scales were refit.
• No tensor was upcast to a higher-precision format.

Created using Bonsai by Prism ML. Built from Qwen3.8-27B (Alibaba Cloud). See NOTICE.txt.

Usage

These files need the PrismML llama.cpp fork, exactly like the original. Stock llama.cpp cannot run
PTQ1_0 or apply the Hadamard activation transform.

# https://github.com/PrismML-Eng/llama.cpp/releases  (tested with prism-b10709-9a9394a, CUDA 12.4)
./llama-server -m Ternary-Bonsai-2-27B-PTQ1_0-abliterated.gguf -ngl 99 -fa on -c 32768

Sampling is the same as the base model: thinking mode temp 1.0, top_p 0.95, top_k 20.

Results

All numbers were measured locally on an RTX 4070 with the fork above.

Evaluation setup

• Harmful prompts: the 104 held-out prompts of mlabonne/harmful_behaviors (test split). Responses
  are labeled by an LLM judge (the original model in non-thinking mode, judge.py):
  • complied: it does what was asked
  • deflected: it answers with a sanitized, fictional or "ethical" substitute
  • refused
  • empty
• KL divergence comes from llama-perplexity --kl-divergence against the original model. The
  harmless chat corpus is the original model's own answers to 200 held-out harmless_alpaca prompts.
• Limits: these are subsets and spot checks, not the 14-benchmark thinking-mode suite reported by
  Prism ML. Coding, instruction following, tool calling, long context and vision were not
  re-evaluated. With 100 GSM8K questions, differences of ±2–3 points are noise.
• Deflections remain: about a quarter of harmful requests still get a watered-down answer. A
  stronger setting reduces this at a measurable KL cost.

Abliteration process

The scripts are in scripts/. The fixed refusal direction is scripts/directions/dirs_proj_both.npy.

1. Why the usual recipe fails on a ternary model

Abliteration replaces every residual-stream writer W with W − α·r·rᵀW. On this model:

• The per-weight change is far below one ternary step (about 1% of a step, at most half a step).
• Re-quantizing the projected matrix therefore just rounds back to the original model, so a naive
  ternary abliteration silently does nothing.
• The alternative is to store the writers at higher precision, which makes the file 60% bigger.

2. Refusal direction (make_dirs.py, tools/actdump.cpp)

• Capture the residual stream (l_out) of every layer at the last prompt token, for 400 harmful and
  400 harmless training prompts.
• Compute the direction at the layer-36 residual:
  • difference of means, harmful minus harmless
  • pooled over both answer positions: <think>\n (thinking mode) and </think>\n\n (no-think
    mode). The two per-mode directions only have cosine 0.64. A direction taken from one mode only
    leaves the other mode broken: the model returns empty answers.
  • orthogonalized against the harmless mean. The raw direction also carries general features;
    removing it wholesale gave KL ≈ 2.
• On held-out prompts it separates harmful from harmless with Cohen's d ≈ 9.6 (thinking) and
  9.05 (no-think).

3. Ternary-preserving ablation (abliterate.py)

• The trits are edited directly. The PTQ1_0 block codec is re-implemented and verified to round-trip
  byte-exact.

• For each writer in layers 16–48, the exact delta −α·r·(rᵀW) is realized by unbiased
  stochastic trit flips, followed by a small greedy correction so that rᵀW′ = (1−α)·rᵀW holds
  column by column.

• Per-component strength:

  • α = 2.0 on attention and linear-attention output writers (attn_output, ssm_out)
  • α = 0.7 on ffn_down

  MLP edits cost far more KL per unit of refusal removed, consistent with
  Heretic's findings.

• The Hadamard rotation of this model is on the input axis only, so an output-side projection is
  exact in the stored basis.

4. Healing (heal.py, tools/wdump.cpp)

• Capture the rotated inputs of every writer (H = E[xxᵀ]) on 16k calibration tokens:

  • alpaca train prompts with the original model's answers
  • GSM8K train thinking traces
  • harmful-prompt train responses
  • wikitext-2 train

  None of these overlap the evaluation sets.

• For each edited writer, refit only the fp16 group scales by ridge least squares, per output row:
  min_s (s·B − w*) H (s·B − w*)ᵀ, where w* = (I − α·r·rᵀ)W is the exact abliterated target.
  Trits, layout and size are unchanged.

5. What didn't work (so you don't have to try)

Reproducing

1. actdump on thinking and no-think prompt files (prompts.py), then make_dirs.py.

2. wdump on the calibration corpus (gen_calib.py), all layers.

3. Build the model:

   python abliterate.py --src Ternary-Bonsai-2-27B-PTQ1_0.gguf --out cand.gguf \
       --dirs directions/dirs_proj_both.npy --dir-layer 36 --layers 16-48 --alpha-attn 2.0 --alpha-ffn 0.7
   python heal.py --orig Ternary-Bonsai-2-27B-PTQ1_0.gguf --edited cand.gguf --out final.gguf --acts wacts \
       --dirs directions/dirs_proj_both.npy --dir-layer 36 --layers 16-48 --alpha-attn 2.0 --alpha-ffn 0.7 --recorrect 0
   

4. Evaluate with evalmodel.sh (refusal + KL), bench.py, mmlu_lp.py and judge.py.

Stochastic flips use seed 0.

Disclaimer

This model has had its refusal behavior removed. It will produce content the original model would
decline to produce. You are responsible for how you use it and for complying with applicable laws.