license: apache-2.0
library_name: gguf
pipeline_tag: text-generation
base_model:

• prism-ml/Ternary-Bonsai-2-27B-gguf
• ProCreations/Ternary-Bonsai-2-27B-MTP
• Qwen/Qwen3.8-27B
  tags:
• gguf
• ternary
• ptq1_0
• mtp
• speculative-decoding
• bonsai
• llama.cpp
• cuda
• local-ai
• 12gb-gpu
• uncensored-optional
• lora
• prism-model-manager

Bonsai 2 27B PTQ1_0 + MTP | 12GB GPU | Uncensored-Ready (Optional OrcaRouter LoRA)

27B-Class Local AI. Built for 12GB GPUs.

Bonsai PTQ1_0 + ProCreations MTP. Up to 91.178 tokens/s in our measured
short-context RTX 4070 Super benchmark with the optimized backend.

A memory-efficient, 27B-class local-inference GGUF built for a single consumer
GPU with roughly 12 GB VRAM. It combines PrismML's PTQ1_0 ternary-quantized
Bonsai 2 backbone with ProCreations' Bonsai-adapted MTP (multi-token prediction)
speculative-decoding head, exported with a Q8_0 prediction head and verified on the
optimized Prism llama.cpp fork with the PR #218 PTQ1_0 CUDA kernels. It is
Uncensored-Ready: an optional, separately-downloaded OrcaRouter rank-1 LoRA can alter
runtime behavior, but the shipped GGUF itself is unchanged and uncensored behavior is
off by default.

Independent community derivative. This repository is an independent, community-made
derivative of work by PrismML and ProCreations. It is not affiliated with,
endorsed by, or maintained by PrismML. The MTP prediction head was trained by
ProCreations, not by the author of this merge. The PTQ1_0 / PR #218 CUDA-optimized
inference path comes from PrismML's llama.cpp fork including PR #218 (developed by
sudoingsx).

At a Glance — Verified Facts

These figures describe different measurements and must not be conflated:

• The 91.178 tok/s short-context MTP 2 number is a single specific benchmark on a
  specific GPU/backend and does not apply to every context length or GPU.
• The GGUF file size (≈5.96 GiB) is not the same as total VRAM usage
  (MTP 2 ≈ 10.7 GB at the tested context); VRAM depends on context, KV cache, MTP
  settings, runtime, and other GPU memory consumers.
• No claim of universally superior speed or efficiency is made without a directly
  comparable benchmark.

About the parameter count shown in Hugging Face search: This repository
contains two GGUF files — the 27.32B-parameter base model (6.4 GB) and a
~2.42M-parameter optional LoRA adapter (9.6 MB). Hugging Face's model indexer
currently picks the adapter file when determining the listed parameter count.
The actual base model has 27,320,697,856 parameters (~27.32B). The LoRA is
a separate runtime adapter that is never merged into the base weights.

📦 Download Prism Model Manager

Jump to Prism Model Manager integration ↓

Manage, load, and benchmark this model (and any GGUF) with the independent
open-source terminal manager for Omarchy / Arch Linux:

• GitHub: https://github.com/Ayshinko/prism-model-manager
• Release v3.0.1 (integrated): https://github.com/Ayshinko/prism-model-manager/releases/tag/v3.0.1

Independent community project — not affiliated with, endorsed by, or maintained by PrismML.

1. What it is and who it's for

Targets local-AI users on a single consumer GPU with roughly 12 GB VRAM who want a
27B-class model with:

• Memory efficiency from PTQ1_0 ternary quantization (backbone fits in a ~6.0 GB
  single file),
• Higher decode throughput from an embedded MTP speculative-decoding head,
• Optional runtime behavior changes (Uncensored-Ready) via a separate OrcaRouter LoRA.

It requires a Prism llama.cpp fork (not stock upstream llama.cpp) to load PTQ1_0
tensors. See Required backend.

2. Model file, size, and checksum

The on-disk GGUF is not an "abliterated" model. Its behavior is the original Bonsai
backbone + ProCreations MTP head. The optional OrcaRouter LoRA is a separate download
and is not merged into this file.

3. Benchmark charts (verified data)

All charts are diagrams computed from the verified benchmark record — they are
not GPU screenshots and are not fabrications.

4. Verified short-context benchmark table

Backend PR #218, context 40960, KV q8_0/q8_0, FA on, -ngl 99, batch 2048, ubatch 512,
Batch-Invariant OFF; 1 warm-up + 3 measured runs, 56-token input → 500 tokens/run.
Median over 3 measured runs.

All responses: 500 predicted tokens, stop limit, 0 CUDA errors, 0 CPU fallback, no early stop.

5. Verified long-context benchmark table

Same backend/settings, 17,531-token input (actual context ≈ 18,031 within a 40,960
window), 500 tokens/run. Median over 3 measured runs.

Do not treat the short-context number as the long-context number: with ~17.5K
tokens resident, every config is ~15–24% slower than its short-context equivalent.

6. Why MTP 2 is the fastest tested draft length

n-max=2 is the fastest in both contexts (91.2 short / 69.1 long tok/s). Going to
n-max=3/4 reduces acceptance (and thus effective speedup) while adding VRAM. In the
table above, MTP1 ≈ +38% short / +33% long over MTP-off; MTP2 ≈ +56% short / +51% long
over MTP-off. For most ~12 GB users draft-mtp n-max=1 remains the recommended
default because it gives the best balance of speed, acceptance, and VRAM headroom.

7. Hardware requirements and VRAM caveats

For an 8 GB GPU: reduced context, different KV quantization, and/or CPU offloading
(-ngl < 99) would be necessary and have not been validated by the published
benchmark.

No FPS, tok/s, VRAM, or compatibility numbers are invented for the RTX 4060, 4060 Ti,
or RTX 4070. Actual VRAM depends on context, KV cache, MTP settings, runtime, and other
GPU memory consumers; always re-validate for your hardware.

Verified VRAM (single-process, idle GPU)

Measured absolute nvidia-smi at context 40960, KV q8_0/q8_0, FA on, -ngl 99. VRAM is
essentially flat during generation, so peak ≈ load.

These are absolute samples in a single-process idle GPU. Other GPUs of the same size,
other drivers/builds, and concurrent workloads will differ.

8. Prism Model Manager 3.0 integration

Prism Model Manager is an
independent, community-developed terminal UI for managing GGUF models with a Prism
backend, built for Omarchy / Arch Linux. It is not an official PrismML
application. It is a general-purpose GGUF model manager (not a Bonsai-only launcher).

Verified capabilities (from the released v3.0.0 source/README):

• Discovering and loading/managing local GGUF models with per-model profiles.
• Selecting a compatible inference backend and validating required flags from its --help.
• Configuring context size, GPU layers, KV cache quantization, Flash Attention, and sampling.
• Controlling MTP speculative decoding (MTP, MTP_MODE, MTP_DRAFT_FLAG,
  MTP_DRAFT_MAX) — v3.0.0 defaults MTP_MODE=draft-mtp mapped to --spec-type draft-mtp.
• Managing local llama-server workflows (start/stop, live logs, status, VRAM/RAM
  monitoring, quick chat tests, raw benchmarks, Web UI).
• Supporting the specialized Prism backend (PTQ1_0/PQ2_0 detection via nvidia-smi).
• Native LoRA support with a per-model "Uncensored LoRA" control (LORA_ENABLED /
  LORA_SCALE / LORA_PATH), wired to the backend's --lora-scaled flag, plus a
  "LoRA Scale Intelligence A/B" benchmark that scores behavior with and without the
  adapter.

Uncensored LoRA   [on]
LoRA scale        [2]
LoRA file         [~/Models/bonsai-abliterate-lora.gguf]

Install and recommended profile

The integrated PMM 3.0.1 release bundles a verified PR218-compatible llama-server — users
don't need to download or build the backend separately. See the manual launch
example for exact llama-server arguments,
and --dry-run / --check in Prism Manager to preview and validate a command before
launching.

# Option 1: integrated release (recommended)
wget https://github.com/Ayshinko/prism-model-manager/releases/download/v3.0.1/prism-model-manager-3.0-linux-x86_64-cuda.tar.gz
tar xzf prism-model-manager-3.0-linux-x86_64-cuda.tar.gz
cd prism-model-manager-3.0-linux-x86_64-cuda
./install.sh
export PATH="$HOME/.local/bin:$PATH"

# Option 2: git clone (source only)
git clone https://github.com/Ayshinko/prism-model-manager.git
cd prism-model-manager && ./install.sh
export PATH="$HOME/.local/bin:$PATH"

First launch (point at your model directory; the backend is pre-configured with the
integrated installer or set PMM_SERVER_BIN manually for git installations):

PMM_MODEL_ROOT="$HOME/Models" \
PMM_SERVER_BIN="$HOME/path/to/prism/llama-server" \
prism-model-manager

Then set these recommended MTP settings for a 12 GB card (validated in v3.0.0 docs and
consistent with the benchmark parameters):

# config.env / TUI values for a 12 GB card
CTX=40960
CTK=q8_0
CTV=q8_0
NGL=99
BATCH=2048
UBATCH=512
FLASH=on
MTP=on
MTP_MODE=draft-mtp       # must be advertised by the backend's --spec-type
MTP_DRAFT_FLAG=--spec-draft-n-max
MTP_DRAFT_MAX=1          # n-max=2 is faster but uses ~0.6 GB more and lower acceptance

The failed to fit params ... n_gpu_layers already set by user to 99, abort startup
warning on MTP configs is the auto-tuner aborting because -ngl 99 is user-forced; it
does not move layers to CPU.

9. Required compatible inference backend

This model requires a Prism llama.cpp fork with the custom PTQ1_0 path and the MTP
(--spec-type draft-mtp) support the model metadata depends on. The stock upstream
llama.cpp release will not load this file's PTQ1_0 tensors. The exact build used for the
benchmarks above is the Prism fork at PR #218 (3443ddece) with the Hadamard-inverse
MTP embedding fix applied.

• The benchmarked PR #218 PTQ1_0 CUDA optimization is not in every public Prism
  build; check the exact revision.
• PR #218 was developed by sudoingsx; it is an upstream open-source contribution
  within the Prism llama.cpp fork, not the author's or this model's work.

10. MTP configuration and runtime requirements

Exact recommended MTP 2 command shape

# llama-server arguments (short-context-verified config, MTP n-max=2)
-c 40960 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -b 2048 -ub 512 \
--spec-type draft-mtp --spec-draft-n-max 2 \
--jinja --reasoning-effort medium

Recommended default for most 12 GB users: --spec-draft-n-max 1 (best speed/VRAM
balance). Use n-max=2 when you want peak short-context speed and have ~1.3 GB headroom.

MTP acceptance varies with prompt/content; speedups are workload-dependent and may not
appear on every hardware/backend revision. Use context ≈ 40960 on a 12 GB card; do not
raise it toward the 262,144 metadata limit (the full-context KV would not fit).

11. Optional OrcaRouter Uncensored LoRA

The main GGUF is not inherently uncensored — it is the original Bonsai backbone +
ProCreations MTP head. Uncensored-related runtime behavior is OPTIONAL and requires
the separate OrcaRouter adapter to be enabled. OrcaRouter is a separate project (not the
author of this repo), and neither the refusal direction nor the adapter was created,
trained, or developed here. We credit it clearly and do not call the default GGUF an
"abliterated" model.

• Adapter source: https://github.com/Continuum-AI-Corp/OrcaBonsai-27B-Uncensored
• Artifact: gguf/bonsai-abliterate-lora.gguf (rank-1 LoRA, F32, 129 sites)
• SHA256: f1669534803d340a496015f5c45125f3437b4d13ec764f40e34488ce83967f42
• License: Apache-2.0 (upstream)
• In this repo: adapters/bonsai-abliterate-lora.gguf — redistributed with attribution under Apache-2.0

The adapter is provided as a convenience download. The main GGUF weights remain
unchanged; the LoRA is applied at runtime as two extra matmuls and is never merged in.

Compatibility status — VERIFIED independently

We independently tested the adapter against this combined PQ1 + MTP GGUF on the PR #218
backend (see ORCAROUTER-COMPAT-REPORT.md):

1. Loads cleanly on this merged GGUF (0 CUDA errors).
2. Genuinely applied in the graph: scale-0 output is byte-identical to the no-LoRA
   baseline, and scale-2 output visibly differs — the documented OrcaRouter discriminator.
3. Coherent output.
4. Works together with MTP: draft-mtp n-max=2 + LoRA generated cleanly at ~82 tok/s.
5. GPU memory and generation speed were measured for each configuration.

Use the native Prism Manager path for the adapter: in Prism Model Manager v3.0.0 you
can enable the per-model "Uncensored LoRA" control (LoRA file → your downloaded
bonsai-abliterate-lora.gguf, scale → 2), and it will pass --lora-scaled automatically
(see section 8). The direct backend-flags manual
commands below remain valid for any setup.

Reproducible startup

# A. main GGUF without LoRA (default, unchanged)
llama-server -m Bonsai-2-27B-PTQ1_0-MTP-Q8_0.gguf -c 40960 -ngl 99 -fa on \
  -ctk q8_0 -ctv q8_0 -b 2048 -ub 512 --jinja --host 127.0.0.1 --port 8080

# B. main GGUF with OrcaRouter LoRA at full strength (scale 1)
llama-server -m Bonsai-2-27B-PTQ1_0-MTP-Q8_0.gguf -c 40960 -ngl 99 -fa on \
  -ctk q8_0 -ctv q8_0 -b 2048 -ub 512 --lora adapters/bonsai-abliterate-lora.gguf --jinja

# C. main GGUF with OrcaRouter LoRA + MTP (draft-mtp, n-max=2)
llama-server -m Bonsai-2-27B-PTQ1_0-MTP-Q8_0.gguf -c 40960 -ngl 99 -fa on \
  -ctk q8_0 -ctv q8_0 -b 2048 -ub 512 --lora adapters/bonsai-abliterate-lora.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 --jinja

For higher ablation strength OrcaRouter documents --lora-scaled adapter.gguf:2 (a sample
of one prompt tells you little; their own eval still saw ~6% refusals at full strength).

When using the Prism Model Manager integrated installer: the bundled PR218 backend is
installed to $PREFIX/lib/prism-llama/. Download adapters/bonsai-abliterate-lora.gguf and
place it in an accessible directory, then set Uncensored LoRA = on and LoRA file to
that path in the PMM settings TUI. The manager passes --lora-scaled <path>:<scale>
automatically.

Enabling a LoRA changes model behavior and may change speed and memory. Use
appropriate, lawful behavioral evaluation and compare with/without. Adapter support here
verified on one backend build; regard this as load/apply/coexistence confirmation, not a
guarantee of any specific behavioral outcome.

12. Manual llama-server launch example

/path/to/prism/llama-server \
  -m /path/to/Bonsai-2-27B-PTQ1_0-MTP-Q8_0.gguf \
  -c 40960 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -b 2048 -ub 512 \
  --spec-type draft-mtp --spec-draft-n-max 1 \
  --jinja --reasoning-effort medium \
  --host 0.0.0.0 --port 8080

13. Compatibility and known limitations

• Requires the Prism llama.cpp fork with PTQ1_0 + MTP support; not stock llama.cpp.
• qwen35 hybrid backbone; quantization variants/backends are not interchangeable.
• MTP acceptance varies with prompt/content; speedups are workload-dependent.
• 12 GB VRAM is the design envelope; not every 12 GB GPU/back-build matches the measured
  throughput. Concurrent workloads reduce headroom, especially at n-max=4.
• The upstream Python gguf library may not parse the custom ternary type tags; use the
  Prism loading path.
• Vision projector (mmproj) and tool-calling are supported by the backend and by Prism
  Model Manager but were not re-validated for this specific merged file.
• Uncensored behavior is optional and off by default; it depends on the separate
  OrcaRouter adapter and was verified only for load/apply/coexistence on one build.

14. Tested hardware/software environment

15. Model provenance and contribution breakdown

Diagram for illustration; see the tables below for exact revisions and checksums.

What the author of this merge contributed:

• Combining the PTQ1_0 backbone with ProCreations' trained MTP head into a single
  Q8_0-head GGUF and applying the compatibility integration (metadata alignment).
• Byte-level per-tensor validation of all 866 tensors and reproducible benchmarks.
• This consumer deployment workflow and documentation.

What the author did NOT do: train the MTP head (ProCreations), invent PTQ1_0
quantization (PrismML), or develop PR #218's CUDA kernels (sudoingsx / PrismML PR #218).

16. License, NOTICE, and credits

This distribution is provided under the Apache License 2.0 (see LICENSE), consistent
with both upstream sources.

• Original model: Bonsai 2 27B by PrismML, Inc. (Apache-2.0). Attribution
  requested: "Created using Bonsai by Prism ML."
• MTP head + integration: ProCreations (Apache-2.0). Fine-tuned the Qwen3.8-27B
  MTP head against frozen Bonsai features; added runtime inverse-rotation handling for the
  MTP embedding lookup. Main Bonsai tensor payloads unchanged.
• Qwen lineage: built from Qwen3.8-27B, Copyright 2026 Alibaba Cloud (Apache-2.0).
• PR #218 CUDA kernels: developed by sudoingsx within the Prism llama.cpp fork.
• OrcaRouter (optional adapter): Continuum-AI-Corp / OrcaRouter (Apache-2.0)
  — refusal-direction ablation LoRA; provided separately, not bundled here.

See NOTICE for the required attribution text.

17. Reproducible benchmark methodology

For every number above the following are fixed and recorded:

• GPU model & VRAM, driver version.
• Backend build id and commit.
• Model file name + SHA256 (section 2).
• Prompt token count (56-token short / 17,531-token long) and generated token count
  (500/run).
• Maximum configured context (40,960) and actual occupied context (556 / 18,031).
• KV cache quantization (q8_0/q8_0), Flash Attention (on), -ngl 99.
• MTP draft length (--spec-draft-n-max N) and GGML_CUDA_BATCH_INVARIANT (off).
• VRAM measurement method (absolute nvidia-smi samples around each request, not
  isolated allocation deltas). Prompt processing reported separately from generation;
  prompt processing speed is never presented as generation speed.
• Number of measured runs (3) and median (not max) reported.
• All runs had zero CUDA errors and zero CPU fallback.

Full per-run detail is in the raw benchmark records (see REPRODUCE.md).

18. Independent community project disclaimer

This is an independent, community-made project. It is not affiliated with, endorsed
by, or maintained by PrismML, ProCreations, Qwen/Alibaba, sudoingsx, the OrcaRouter
team, or Omarchy. Brands, models, and code remain the property of their respective
owners and keep their own licenses. Nothing here is provided with warranty; validate
everything for your own hardware.