license: apache-2.0
library_name: mlx
pipeline_tag: image-text-to-text
base_model:

• 3hs4n/Ternary-Bonsai-2-27B-Uncensored-Heretic-v2-mlx-2bit
  tags:
• mlx
• mlx-vlm
• ternary
• 2-bit
• bonsai
• heretic
• uncensored
• conversational
• reasoning
• thinking
• image-text-to-text
• tool-calling
• apple-silicon
• speculative-decoding
• dflash2
• draft-model

Ternary Bonsai 2 27B Uncensored Heretic v2: MLX 2-bit with DFlash2

A self-contained folder that serves the original Heretic v2 MLX pack with DFlash2 speculative decoding. It holds the original pack unchanged, including its full-precision (FP16) vision tower (model.safetensors is byte-identical, 8,595,478,392 bytes), the pinned BF16 DFlash2 drafter (3.85 GB, Apache-2.0), a small runtime that adds the drafter to native mlx-vlm 0.7.2, and the base pack's plain serve.py. Nothing is requantized or retrained.

What it does. Greedy output is the same token sequence as plain decoding. The drafter only proposes tokens, the target verifies them with single-token decode arithmetic, and rejected positions are rolled back. DFlash2 changes speed, not answers.

Performance. On the tested Apple M1 Max, this Original pack measured 1.07-1.23x plain decode speed for code and math, 1.16x for tool calls, and 0.73-0.95x for chat, prose, thinking, vision and long documents. The median over 13 prompts was 0.90x, so general chat is faster with plain decoding. DFlash2 needs about 4 GiB more memory for short prompts, growing with prompt length. Requests with temperature 0 use speculation; sampled requests run plain. The Compact sibling pack measured a 0.96x median in its separate run. This pack's own measurements are under Results.

The Original shares Compact's language weights byte for byte. The two Heretic v2 packs differ only in the vision tower: this pack keeps it in FP16, while Compact quantizes 27 vision MLP expansion layers. Because the language tensors and the Hadamard manifest are identical, draft acceptance on text should match the Compact pack's. That is an expectation; the Results and Correctness sections hold this pack's own measurements, and figures marked as the Compact sibling come from its card.

Recommendation. Use this pack for code- and math-heavy greedy work. For general chat, long documents, long contexts and sampling, use the plain Original base pack. This folder also contains serve.py, so the plain server is always available from the same download.

Which pack to use

What is in this folder

Omitted from the original base: lmstudio-model.yaml. Replaced or extended: README.md, NOTICE.txt, SHA256SUMS. Everything else is as in the base pack. The public test fixtures used by the DFlash2 fixture check ship with the Compact pack, not with this one.

Requirements

• An Apple Silicon Mac. DFlash2 uses a custom Metal kernel and MLX; other platforms are not supported.
• Python 3.11 (the base packs were tested with 3.11) and the pinned packages in requirements.txt. The runtime requires mlx 0.32.2 and mlx-vlm 0.7.2 exactly. It checks that the classes and functions it adjusts exist, fingerprints the verifier's _attention source (and DFlashAttention.__call__ when the window mask is on), and refuses to start on other versions or when those sources differ from the audited ones.
• About 12.4 GB of disk for the weights (8.60 GB target plus 3.85 GB drafter).
• Unified memory: DFlash2 needs about 4 GiB more than plain decoding for short prompts (measured on the Compact sibling), and more as prompts grow. The RAM tiers in the base card are estimates for plain decoding. Peak memory measured on this pack is in Results.

Quick start

Create an environment, download the pack and install the pinned dependencies:

python3.11 -m venv .venv-heretic
source .venv-heretic/bin/activate
python -m pip install huggingface-hub==1.33.0
hf download 3hs4n/Ternary-Bonsai-2-27B-Uncensored-Heretic-v2-mlx-2bit-DFlash2 \
  --local-dir Ternary-Bonsai-2-27B-Uncensored-Heretic-v2-mlx-2bit-DFlash2
cd Ternary-Bonsai-2-27B-Uncensored-Heretic-v2-mlx-2bit-DFlash2
python -m pip install -r requirements.txt

Start the DFlash2 server from inside the folder:

python serve_dflash2.py --host 127.0.0.1 --enable-thinking

The default API address is http://127.0.0.1:8080/v1, the native mlx-vlm server default. Use --port to choose another free port. The server listens on 127.0.0.1 only; --host accepts only 127.0.0.1 or localhost. The public model ID is the folder name, Ternary-Bonsai-2-27B-Uncensored-Heretic-v2-mlx-2bit-DFlash2 when you download as above; read /v1/models to see it, or set it with --model-id. The first speculative request pays Metal kernel compilation, and each new verify row count pays it once more, so time a repeated request.

--enable-thinking makes thinking the default, as in the base card; a request can turn it off with "enable_thinking": false.

Send "temperature": 0 to use speculative decoding:

curl -i http://127.0.0.1:8080/v1/chat/completions -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"Write a Python function that reverses a string."}],
       "temperature":0,"enable_thinking":false,"max_tokens":200}'

The response header x-heretic-dflash2 says speculative or fallback; reason=<reason>. Speculative responses carry draft_rounds, draft_n and draft_n_accepted in timings. The server logs each request's route, mode and reason when it arrives, and its acceptance and peak memory when it finishes.

The plain server stays available from the same folder, with the same model and no drafter:

python serve.py \
  --model . \
  --model-id Ternary-Bonsai-2-27B-Uncensored-Heretic-v2-mlx-2bit \
  --host 127.0.0.1 \
  --enable-thinking

Both servers default to port 8080, so run one at a time or give one of them --port. python serve_dflash2.py --host 127.0.0.1 --no-dflash2 serves the DFlash2 wrapper with the same settings and no drafter, for like-for-like comparisons. Image requests use OpenAI image_url content parts and tool requests use an OpenAI tools array, as described in the base card.

Server options

python serve_dflash2.py --help lists every option. The DFlash2 settings:

At launch the drafter's config, README, safetensors header and size are checked, and the target's size and configuration files are checked against their pins. SHA256SUMS covers the full weight hashes; see Reproduce and verify.

Extra mlx-vlm options pass through only by exact name: --enable-thinking, --log-level, --max-tokens, --prefill-step-size, --thinking-start-token, --thinking-end-token and --top-logprobs-k. Any other option, abbreviation or = form of a refused name is an error. Inherited environment variables that change generation, caching or batching stop the server with a list of names, in the baseline mode too: EXPERT_CACHE_GB, KV_BITS, KV_GROUP_SIZE, KV_KEY_BITS, KV_KEY_SCHEME, KV_QUANT_SCHEME, KV_VALUE_BITS, KV_VALUE_SCHEME, MAX_KV_SIZE, MLX_TRUST_REMOTE_CODE, MLX_VLM_ENABLE_THINKING, MLX_VLM_MAX_NUM_SEQS, MLX_VLM_MAX_TOKENS, MLX_VLM_RESPONSE_STORE_LIMIT, MLX_VLM_SPEC_BATCH_COALESCE_MS, MLX_VLM_THINKING_BUDGET, MLX_VLM_THINKING_END_TOKEN, MLX_VLM_THINKING_START_TOKEN, MLX_VLM_VISION_CACHE_SIZE, PREFILL_STEP_SIZE, QUANTIZED_KV_START, TOP_LOGPROBS_K, and any APC_* except APC_ENABLED=0. Unset them if the server refuses to start.

Request routing

The server is single-sequence with prefix caching off. Requests that speculative decoding cannot serve with identical output run plain or are refused, and the response header says which:

A request that omits temperature is served plain with the pack's sampling defaults, so always send "temperature": 0 to get the speedup. Routing and refusals apply only when the drafter is loaded.

How it works

Each round the drafter reads the target's hidden states at layers 5, 19, 33, 47 and 61, proposes a block of tokens in one forward pass, and the target checks the whole block in one forward pass. Accepted tokens are kept, the target's own token replaces the first rejected one, and the target cache is rolled back to that point: attention KV is trimmed and the GatedDeltaNet recurrent and convolution states are restored from per-position records. This is mlx-vlm's DFlash path, from the method in DFlash: Block Diffusion for Flash Speculative Decoding. heretic_dflash2.py adjusts it at runtime without editing installed packages:

• Exact verification. During verification the 401 Hadamard-packed linears (lm_head included) use heretic_dflash2_kernels.py, a Metal kernel that reproduces MLX's single-row 2-bit arithmetic for up to 8 rows at once and reads the float16 scales directly. The dense GatedDeltaNet gate projections and 2-row attention run one position at a time. Stock mlx-vlm 0.7.2 batches them, which in an on-device check made verify logits differ slightly from single-token decode, and stock batched quantized matmuls stopped matching single-row results above 5 rows. Prefill and plain decoding are untouched.
• BF16 drafter activations. The drafter's inputs are cast to bfloat16 and its output back to float32 for the target head. Without this, the float32 target activations promote the BF16 drafter to float32 and upcast its weights on every call.
• Options kept for experiments. A fixed block, the trained candidate selector for greedy drafting (stock mlx-vlm skips it; the z-lab and reference runtimes use it), and the z-lab sliding-window mask. None was measurably faster on the Compact sibling, where the defaults were chosen; see its card for the ablations.

Why chat is slower: a round costs more than a plain decode step, because verifying several rows exactly is limited by arithmetic, so a round has to emit enough tokens to pay for itself. On the Compact sibling, code, math and tool-call cases emitted 4.0-4.7 tokens per round, while chat and prose emitted about 2.

Correctness

DFlash2 is built so that greedy output equals plain decoding of the same pack: verification uses single-token decode arithmetic, and rollback restores the attention and recurrent state exactly. The checks that were run on this pack follow. The bitwise checks cover specific prompts and shapes; they are tests, not a proof for every input. Speculation runs at batch size 1.

Bitwise parity (Metal kernel against stock single-row matmul, one verify forward against single-token decode steps, and rollback after partial acceptance):

The test output is in evidence/dflash2-parity.txt.

Pack fixtures (public synthetic text, thinking, tool-call, colour, spatial, counting, OCR and chart cases, run under plain decoding and under DFlash2; the fixtures ship with the Compact pack):

Default settings (adaptive block, argmax drafting, exact-kernel verify). Details: evidence/dflash2-fixture-checks.md and .json.

This is a targeted regression check, not a vision or reasoning benchmark.

Results

This section reports measurements of this Original pack. The Compact pack's measurements are in its card and are not repeated here.

Apple M1 Max, 64 GiB, macOS 27.0.1, mlx 0.32.2, mlx-vlm 0.7.2. Greedy decoding, default DFlash2 settings, prefill step 512, 13 public cases, 3 repeats after 1 warmup pass per process; each arm and repeat in a fresh process, interleaved and rotated. Every measured case started with 1-minute load below 4 and 5-minute load below 5; observed maxima were 3.87 and 3.33. Decode tok/s is (tokens - 1) / (last token time - first token time).

The screen was locked and AC power connected throughout all 245 active-worker condition samples, taken about every 10 seconds. No lock, power-source or raw power-mode change was observed. Unlocked performance was not measured. See lock and power evidence. The largest plain-repeat deviation from its case median was 1.67%, below the 8% release limit.

Median speedup 0.90x, worst case 0.73x. Draft acceptance 71%, 2.71 tokens per round. 13/13 cases token-identical to plain decoding in every repeat.

Raw summaries: evidence/dflash2-bench-default-vs-plain.md and .json.

Reproduce and verify

Integrity of this folder, from inside it:

shasum -a 256 -c SHA256SUMS

dflash2-pack.json records the base and drafter repositories, revisions, sizes and sha256 values, so you can compare them with the Hugging Face pages. python serve_dflash2.py --dflash2-full-hash rehashes the target and drafter weights at launch.

To compare speed on your own machine, close other heavy applications first (background load lowers both arms), start the server with and without --no-dflash2, and send the same temperature 0 requests to each. The outputs should match exactly; compare wall-clock time on code or math prompts and on chat prompts.

The benchmark runner, the fixture checker and the bitwise parity tests that produced the numbers in this card live in the source repository's scripts and tests, not in this folder. Their raw outputs are the dflash2-* files in evidence/. The pack does not ship the benchmark prompt suite.

Limitations

• Speed depends on the content and the machine. Chat, prose, long documents and thinking were slower than plain decoding on the tested M1 Max, because the drafter's guesses are accepted less often there. Check this pack's own numbers under Results.
• Memory. About 4 GiB more for short prompts in this run, growing with prompt length (+8.8 GiB at 33k tokens on the Compact sibling). Prefer plain decoding for long contexts. Keeping only the last 2,047 positions of the drafter taps would remove most of the growth; it is not implemented.
• Greedy only. Sampling, logprobs, penalties and logit_bias run plain; thinking budgets, structured output, n > 1 and the Anthropic Messages API are refused.
• Single sequence, no prefix cache. Concurrent requests queue, and every turn of a conversation prefills its whole prompt again.
• Drafter fit. The drafter was fine-tuned on the base Ternary Bonsai 2 27B, not on Heretic v2. A Heretic-specific drafter might accept more tokens; no training was done here.
• Pinned runtime. The runtime requires mlx 0.32.2 and mlx-vlm 0.7.2 exactly and refuses to start on other versions, missing patch targets or changed attention source.
• No approximate fast path. A float16 matrix-unit verify, as in the drafter's reference runtime, would be faster but not bit-exact, so it is not included.
• Launch-time checks cover the drafter's metadata and size; use --dflash2-full-hash to rehash the weights.
• Vision. Image requests are speculative by default. This pack's image cases measured 0.89-0.90x of plain decode speed; --dflash2-multimodal plain routes images, audio and video to plain decoding.
• Not covered. Other Mac models, contexts near the configured 262,144-token limit, and sampled decoding quality were not tested for DFlash2.

serve_dflash2.py publishes the fixed model ID and replaces known model, drafter, home and working-directory paths in runtime metadata, response headers, error responses and logs, as the base serve.py does. It does not remove private content supplied in prompts, generated answers or tool results.

Intended use, credits and license

OS-Software describes Heretic v2 as a decensored derivative with an OT-Ridge LoRA merged into the ternary language weights. Its publisher recommends research and experimentation, including alignment evaluation and red-teaming, and advises against public end-user service deployment. DFlash2 does not change the language weights or restore safety alignment; because output is identical to plain decoding, it carries the same reduced alignment. Outputs can be inaccurate, biased, offensive or harmful; the word "uncensored" does not establish universal refusal-free behavior.

• DFlash: the speculative-decoding method, from DFlash: Block Diffusion for Flash Speculative Decoding (arXiv 2602.06036).
• z-lab: publisher of z-lab/Qwen3.8-27B-DFlash2, the DFlash2 drafter that naklitechie fine-tuned for Ternary Bonsai 2; its sliding-window mask is available here as an option.
• naklitechie: Qwen3.8-27B-DFlash2-ternary-bonsai2, the BF16 drafter fine-tuned on Ternary Bonsai 2 27B, included here unchanged (Apache-2.0).
• OS-Software: Heretic v2 language-weight modification and source GGUF publication (Heretic v2 GGUF).
• Prism ML: Bonsai 2 ternary model, Hadamard-aware representation, original MLX vision pack and compatible runtime work (Bonsai 2 MLX pack). Created using Bonsai by Prism ML.
• Qwen / Alibaba Cloud: underlying Qwen lineage, identified by the retained notice as Qwen3.8-27B.
• p-e-w: Heretic, acknowledged by the source publisher.
• Bonsai demo: conversion and runtime reference project.
• 3hs4n: community native MLX packaging and this DFlash2 pack: the runtime files and the measurements in this card.

Distributed under Apache 2.0. The original LICENSE and attribution in NOTICE.txt are retained; NOTICE.txt gains a section describing the DFlash2 add-on. The target weights are unchanged from the original base, the drafter is included unchanged under its own Apache-2.0 license, and the changes in this folder are the drafter, the runtime files (heretic_dflash2.py, heretic_dflash2_kernels.py, serve_dflash2.py), the evidence files, dflash2-pack.json, this card and the omission of lmstudio-model.yaml. The original MLX release's packaging and server-helper changes are retained. Source-publisher use guidance is attributed above; redistribution terms are governed by the license.