license: apache-2.0
library_name: mlx
pipeline_tag: image-text-to-text
base_model:

• 3hs4n/Ternary-Bonsai-2-27B-Uncensored-Heretic-v2-mlx-2bit-compact
  tags:
• mlx
• mlx-vlm
• ternary
• 2-bit
• bonsai
• heretic
• uncensored
• conversational
• reasoning
• thinking
• vision
• image-text-to-text
• tool-calling
• apple-silicon
• speculative-decoding
• dflash2
• draft-model

Ternary Bonsai 2 27B Uncensored Heretic v2: Compact MLX 2-bit with DFlash2

A self-contained folder that serves the Compact Heretic v2 MLX pack with DFlash2 speculative decoding. It holds the Compact pack unchanged (model.safetensors is byte-identical, 8,365,392,904 bytes), the pinned BF16 DFlash2 drafter (3.85 GB, Apache-2.0), a small runtime that adds the drafter to native mlx-vlm 0.7.2, and the base pack's plain serve.py. Nothing is requantized or retrained.

What it does. Greedy output is the same token sequence as plain decoding. The drafter only proposes tokens, the target verifies them with single-token decode arithmetic, and rejected positions are rolled back. DFlash2 changes speed, not answers.

What it does not do. It is not faster for everything. On the tested Apple M1 Max it is faster for code, math and tool calls (1.07-1.24x) and slower for chat, prose, thinking, vision and long documents (0.74-0.96x). Over the 13 test prompts the median speedup is 0.96x, which is slightly slower than plain decoding. It needs about 4 GiB more memory than plain decoding, and the difference grows with prompt length. It is greedy-only: only requests with temperature 0 are speculative.

Recommendation. Use this pack for code- and math-heavy greedy work. For general chat, long documents, long contexts and sampling, use the plain Compact base pack. This folder also contains serve.py, so the plain server is always available from the same download.

Decode speed, greedy, Apple M1 Max, default settings. Each row averages the cases of its category (per-case medians of two interleaved repeats); the speedup range is the lowest and highest case in the category. Every output was token-identical to plain decoding. Full per-case numbers are under Results.

Which pack to use

Compact quantizes 27 vision MLP expansion layers from FP16 to native MLX affine 2-bit/group-128. That is lossy for image understanding and OCR; the language tensors are identical to the Original pack. See the Compact base card for that trade-off.

What is in this folder

Omitted from the Compact base: lmstudio-model.yaml. Replaced or extended: README.md, NOTICE.txt, SHA256SUMS. Everything else is as in the base pack.

Requirements

• An Apple Silicon Mac. DFlash2 uses a custom Metal kernel and MLX; other platforms are not supported.
• Python 3.11 (tested with 3.11.15) and the pinned packages in requirements.txt. The runtime requires mlx 0.32.2 and mlx-vlm 0.7.2 exactly. It checks that the classes and functions it adjusts exist, fingerprints the verifier's _attention source (and DFlashAttention.__call__ when the window mask is on), and refuses to start on other versions or when those sources differ from the audited ones.
• About 12.2 GB of disk for the weights (8.37 GB target plus 3.85 GB drafter).
• Unified memory: measured peak MLX memory on the 13-prompt suite was 8.4-11.6 GiB for plain decoding and 12.3-15.9 GiB with DFlash2, on a 64 GiB machine. The RAM tiers in the Compact base card are estimates for plain decoding; DFlash2 adds the drafter and more as prompts grow. Nothing was tested below 64 GiB.

Quick start

Create an environment, download the pack and install the pinned dependencies:

python3.11 -m venv .venv-heretic
source .venv-heretic/bin/activate
python -m pip install huggingface-hub==1.33.0
hf download 3hs4n/Ternary-Bonsai-2-27B-Uncensored-Heretic-v2-mlx-2bit-compact-DFlash2 \
  --local-dir Ternary-Bonsai-2-27B-Uncensored-Heretic-v2-mlx-2bit-compact-DFlash2
cd Ternary-Bonsai-2-27B-Uncensored-Heretic-v2-mlx-2bit-compact-DFlash2
python -m pip install -r requirements.txt

Start the DFlash2 server from inside the folder:

python serve_dflash2.py --host 127.0.0.1 --enable-thinking

The default API address is http://127.0.0.1:8080/v1, the native mlx-vlm server default. Use --port to choose another free port. The server listens on 127.0.0.1 only; --host accepts only 127.0.0.1 or localhost. The public model ID is the folder name, Ternary-Bonsai-2-27B-Uncensored-Heretic-v2-mlx-2bit-compact-DFlash2 when you download as above; read /v1/models to see it, or set it with --model-id. The first speculative request pays Metal kernel compilation, and each new verify row count pays it once more, so time a repeated request.

--enable-thinking makes thinking the default, as in the base card; a request can turn it off with "enable_thinking": false.

Send "temperature": 0 to use speculative decoding:

curl -i http://127.0.0.1:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"Write a Python function that reverses a string."}],
       "temperature":0,"enable_thinking":false,"max_tokens":200}'

The response header x-heretic-dflash2 says speculative or fallback; reason=<reason>. Speculative responses carry draft_rounds, draft_n and draft_n_accepted in timings. The server logs each request's route, mode and reason when it arrives, and its acceptance and peak memory when it finishes.

The plain server stays available from the same folder, with the same model and no drafter:

python serve.py \
  --model . \
  --model-id Ternary-Bonsai-2-27B-Uncensored-Heretic-v2-mlx-2bit-compact \
  --host 127.0.0.1 \
  --enable-thinking

Both servers default to port 8080, so run one at a time or give one of them --port. python serve_dflash2.py --host 127.0.0.1 --no-dflash2 serves the DFlash2 wrapper with the same settings and no drafter, for like-for-like comparisons. Image requests use OpenAI image_url content parts and tool requests use an OpenAI tools array, as described in the base card.

Server options

python serve_dflash2.py --help lists every option. The DFlash2 settings:

At launch the drafter's config, README, safetensors header and size are checked, and the target's size and configuration files are checked against their pins. SHA256SUMS covers the full weight hashes; see Reproduce and verify.

Extra mlx-vlm options pass through only by exact name: --enable-thinking, --log-level, --max-tokens, --prefill-step-size, --thinking-start-token, --thinking-end-token and --top-logprobs-k. Any other option, abbreviation or = form of a refused name is an error. Inherited environment variables that change generation, caching or batching stop the server with a list of names, in the baseline mode too: EXPERT_CACHE_GB, KV_BITS, KV_GROUP_SIZE, KV_KEY_BITS, KV_KEY_SCHEME, KV_QUANT_SCHEME, KV_VALUE_BITS, KV_VALUE_SCHEME, MAX_KV_SIZE, MLX_TRUST_REMOTE_CODE, MLX_VLM_ENABLE_THINKING, MLX_VLM_MAX_NUM_SEQS, MLX_VLM_MAX_TOKENS, MLX_VLM_RESPONSE_STORE_LIMIT, MLX_VLM_SPEC_BATCH_COALESCE_MS, MLX_VLM_THINKING_BUDGET, MLX_VLM_THINKING_END_TOKEN, MLX_VLM_THINKING_START_TOKEN, MLX_VLM_VISION_CACHE_SIZE, PREFILL_STEP_SIZE, QUANTIZED_KV_START, TOP_LOGPROBS_K, and any APC_* except APC_ENABLED=0. Unset them if the server refuses to start.

Request routing

The server is single-sequence with prefix caching off. Requests that speculative decoding cannot serve with identical output run plain or are refused, and the response header says which:

A request that omits temperature is served plain with the pack's sampling defaults, so always send "temperature": 0 to get the speedup. Routing and refusals apply only when the drafter is loaded.

How it works

Each round the drafter reads the target's hidden states at layers 5, 19, 33, 47 and 61, proposes a block of tokens in one forward pass, and the target checks the whole block in one forward pass. Accepted tokens are kept, the target's own token replaces the first rejected one, and the target cache is rolled back to that point: attention KV is trimmed and the GatedDeltaNet recurrent and convolution states are restored from per-position records. This is mlx-vlm's DFlash path, from the method in DFlash: Block Diffusion for Flash Speculative Decoding. heretic_dflash2.py adjusts it at runtime without editing installed packages:

• Exact verification. During verification the 401 Hadamard-packed linears (lm_head included) use heretic_dflash2_kernels.py, a Metal kernel that reproduces MLX's single-row 2-bit arithmetic for up to 8 rows at once and reads the float16 scales directly. The dense GatedDeltaNet gate projections and 2-row attention run one position at a time. Stock mlx-vlm 0.7.2 batches them, which in an on-device check made verify logits differ slightly from single-token decode, and stock batched quantized matmuls stopped matching single-row results above 5 rows. Prefill and plain decoding are untouched.
• BF16 drafter activations. The drafter's inputs are cast to bfloat16 and its output back to float32 for the target head. Without this, the float32 target activations promote the BF16 drafter to float32 and upcast its weights on every call.
• Options kept for experiments. A fixed block, the trained candidate selector for greedy drafting (stock mlx-vlm skips it; the z-lab and reference runtimes use it), and the z-lab sliding-window mask. None was measurably faster on this machine; see Ablations.

Why chat is slower: a round costs more than a plain decode step, because verifying several rows exactly is limited by arithmetic, so a round has to emit enough tokens to pay for itself. In the suite, code, math and tool-call cases emitted 4.0-4.7 tokens per round, while chat and prose emitted about 2.

Correctness

The bitwise checks cover these prompts and shapes; they are tests, not a proof for every input. Speculation runs at batch size 1.

The fixture check ran the 20 public synthetic cases in fixtures/ (same scoring as validate_heretic_compact.py) under plain decoding and under the default DFlash2 configuration: plain 20/20, DFlash2 20/20, token-identical 20/20, 65 speculative rounds (tool round trips included) at 2.92 emitted tokens per round, effective verify mode exact-kernel. Thinking cases used a plain 384-token limit, because thinking_budget is refused with a drafter. The per-case table is in evidence/dflash2-fixture-checks.md. This is a targeted regression check, not a vision or reasoning benchmark.

Results

Apple M1 Max, 64 GiB, macOS 27.0.1, Python 3.11.15, mlx 0.32.2, mlx-vlm 0.7.2, transformers 5.14.1. Greedy decoding, prefill step 512, seed 98243, 13 public cases (code, math, chat, prose, thinking, two long documents, two images from the pack's fixtures, a tool call). Two repeats after one warmup pass per process; each arm and repeat ran in a fresh process, interleaved with the plain arm and rotated in order; 1-minute load averages were 2.35-6.43 in this run. Decode tok/s is (tokens - 1) / (last token time - first token time). Plain decoding ran at 20.8-21.1 tok/s on short prompts. Peak memory is MLX's peak after a reset before each case, and includes the drafter weights.

Default configuration per case

Median speedup 0.96x; the slowest case is 0.74x. Draft acceptance 71%; per draft position 87%, 70%, 57% and 49% (mean over the 13 cases). Per repeat, the adaptive schedule ran 539 rounds at block 3, 161 at block 5 and 1 at block 2. Time to first token stayed within about 5% of plain decoding, because prefill is the same code path. All 13 cases were token-identical to plain decoding in every repeat.

Ablations

Alternatives to the default, with the same suite on the same machine:

All 13 cases were token-identical to plain decoding in every arm and repeat. Findings:

• Target verification dominates the round. On this GPU the exact 2-bit matmul is limited by arithmetic. The exact kernel makes the same arithmetic about 16% faster end to end than stock MLX (0.91x vs 0.79x at block 5).
• Larger blocks accept more tokens per round but cost more to verify: block 8 beats plain decoding only on code, math and the tool call, and beats block 5 on a single case. The adaptive 3..5 schedule shrinks the block when drafts fail, which helps chat and long documents most.
• The trained selector costs about 14 ms per round (12-15 ms per case, comparing the window-mask selector arm with block-5 argmax in the same ablation run; estimated from tokens per round and decode speed). At blocks of 3-5 it barely changes acceptance, so plain argmax drafting was faster.
• The window mask did not change acceptance on these prompts.

Memory and long prompts

This test used the earlier default (block 5, selector), one repeat and 64 generated tokens:

The drafter weights are 3.6 GiB. The rest grows by about 0.18 GiB per 1,000 prompt tokens, because mlx-vlm keeps the drafter's float32 hidden-state taps for the whole prompt, although the drafter attends only to the last 2,047 positions. The current default at 7,583 prompt tokens peaked at 15.9 GiB against 11.6 GiB plain, in line with the first row. Outputs were token-identical in all three cases. For long contexts, prefer plain decoding.

Reproduce and verify

Integrity of this folder, from inside it:

shasum -a 256 -c SHA256SUMS

dflash2-pack.json records the base and drafter repositories, revisions, sizes and sha256 values, so you can compare them with the Hugging Face pages. python serve_dflash2.py --dflash2-full-hash rehashes the target and drafter weights at launch.

To compare speed on your own machine, close other heavy applications first (background load lowers both arms), start the server with and without --no-dflash2, and send the same temperature 0 requests to each. The outputs should match exactly; compare wall-clock time on code or math prompts and on chat prompts.

The benchmark runner, the fixture checker and the bitwise parity tests that produced the numbers above live in the source repository's scripts and tests, not in this folder. They used the settings listed under Results, and their raw outputs are the dflash2-* files in evidence/. The pack does not ship the benchmark prompt suite.

Limitations

• Speed depends on the content and the machine. Measured on one M1 Max only. Chat, prose, long documents and thinking are slower than plain decoding, because the drafter's guesses are accepted less often there.
• Memory. About 4 GiB more for short prompts, growing with prompt length (+8.8 GiB at 33k tokens). Prefer plain decoding for long contexts. Keeping only the last 2,047 positions of the drafter taps would remove most of the growth; it is not implemented.
• Greedy only. Sampling, logprobs, penalties and logit_bias run plain; thinking budgets, structured output, n > 1 and the Anthropic Messages API are refused.
• Single sequence, no prefix cache. Concurrent requests queue, and every turn of a conversation prefills its whole prompt again.
• Drafter fit. The drafter was fine-tuned on the base Ternary Bonsai 2 27B, not on Heretic v2. A Heretic-specific drafter might accept more tokens; no training was done here.
• Pinned runtime. The runtime requires mlx 0.32.2 and mlx-vlm 0.7.2 exactly and refuses to start on other versions, missing patch targets or changed attention source.
• No approximate fast path. A float16 matrix-unit verify, as in the drafter's reference runtime, would be faster but not bit-exact, so it is not included.
• Launch-time checks cover the drafter's metadata and size; use --dflash2-full-hash to rehash the weights.
• Vision. Image requests are speculative by default and measured slightly slower than plain decoding (0.93-0.96x). Compact's vision quantization is lossy; accuracy of images and OCR has not been benchmarked beyond the targeted fixtures.
• Not tested. Memory below 64 GiB, other Mac models, contexts near the configured 262,144-token limit, and sampled decoding quality.

serve_dflash2.py publishes the fixed model ID and replaces known model, drafter, home and working-directory paths in runtime metadata, response headers, error responses and logs, as the base serve.py does. It does not remove private content supplied in prompts, generated answers or tool results.

Intended use, credits and license

OS-Software describes Heretic v2 as a decensored derivative with an OT-Ridge LoRA merged into the ternary language weights. Its publisher recommends research and experimentation and advises against public end-user service deployment. DFlash2 does not change the language weights or restore safety alignment; because output is identical to plain decoding, it carries the same reduced alignment. Outputs can be inaccurate, biased, offensive or harmful; the word "uncensored" does not establish universal refusal-free behavior.

• DFlash: the speculative-decoding method, from DFlash: Block Diffusion for Flash Speculative Decoding (arXiv 2602.06036).
• z-lab: publisher of z-lab/Qwen3.8-27B-DFlash2, the DFlash2 drafter that naklitechie fine-tuned for Ternary Bonsai 2; its sliding-window mask is available here as an option.
• naklitechie: Qwen3.8-27B-DFlash2-ternary-bonsai2, the BF16 drafter fine-tuned on Ternary Bonsai 2 27B, included here unchanged (Apache-2.0).
• OS-Software: Heretic v2 language-weight modification and source GGUF publication.
• Prism ML: Bonsai 2 ternary model, Hadamard-aware representation, original MLX vision pack and compatible runtime work. Created using Bonsai by Prism ML.
• Qwen / Alibaba Cloud: underlying Qwen lineage, identified by the retained notice as Qwen3.8-27B.
• p-e-w: Heretic, acknowledged by the source publisher.
• Bonsai demo: conversion and runtime reference project.
• 3hs4n: community native MLX packaging, the Compact vision quantization, and this DFlash2 pack: the runtime files and the measurements above.

Distributed under Apache 2.0. The original LICENSE and attribution in NOTICE.txt are retained; NOTICE.txt gains a section describing the DFlash2 add-on. The target weights are unchanged from the Compact base, the drafter is included unchanged under its own Apache-2.0 license, and the changes in this folder are the drafter, the runtime files (heretic_dflash2.py, heretic_dflash2_kernels.py, serve_dflash2.py), the evidence files, dflash2-pack.json, this card and the omission of lmstudio-model.yaml. Source-publisher use guidance is attributed above; redistribution terms are governed by the license.