license: other
license_name: tencent-hy-license
license_link: LICENSE
language:
- en
- zh
pipeline_tag: text-generation
library_name: mlx
base_model: orcarouter/Hy4-flash-abliterated
base_model_relation: quantized
tags: - mlx
- apple-silicon
- hunyuan
- hy4-flash
- moe
- abliterated
- uncensored
- mixed-precision
- gptq
Hy4-flash-abliterated-MLX
|
🛡️ New from Orca — OrcaCyber Zero 1.5 Post-trained for cybersecurity: vulnerability analysis, vulnerability reproduction, red teaming & offensive-security research. 98.01% pass@1 on CyberGym Level 1 · 1M context · function calling · OpenAI-compatible API. |
|
🔓 All hosted cyber and abliterated models as API Skip the download: OrcaRouter serves its cybersecurity and abliterated models behind one OpenAI-compatible API key. |
|
🔌 New from Orca — Orca Connect Running this model on your own Mac? Orca Connect puts the machine on your desk behind your usual OrcaRouter API key, next to 200+ hosted models, reachable from a laptop, a phone or a cloud agent. Outbound only: no port forwarding, no tunnel, no firewall rule. Requests your own hardware serves cost $0, and |
Abliterated Hy4-flash (220B MoE) in MLX, for 128 GB and 256 GB Macs.
GPTQ in stock MLX affine quantization · ~86 GiB or ~156 GiB of weights in memory · the Engram tables stay on disk
OrcaRouter AI Gateway · X · Discord · GitHub · All Models
Abliterated Hy4-flash, run locally on Apple Silicon.
These are MLX builds of
orcarouter/Hy4-flash-abliterated,
the abliterated (refusal-reduced)tencent/Hy4-flash.
Hy4-flash is a 220B MoE with 11.3B active parameters per token: 42 layers (1 dense, 41
latent-MoE with 512 experts, 16 active), hybrid sparse / sliding-window MLA attention, four
hyper-connection streams, Engram n-gram memory at four layers, and a 1M context.There are two builds, one for a 128 GB Mac and one for a 256 GB Mac, both quantized with
GPTQ onto MLX's own affine grid, so MLX runs them with its stock quantized kernels. The
model code ships in each folder ashy4_mlx/, the MLX port of Hy4-flash by
Xylolabs.Sized by what stays in memory. 63B of the 220B parameters are the Engram hash tables, and a
token reads only a few of their rows.hy4_mlxreads them row by row from disk through a memory
map, so what the GPU must hold is the rest: about 86 GiB for the 128GB build, inside the
~96 GiB a 128 GB Mac lets the GPU wire by default, and about 156 GiB for the 256GB build,
inside the ~192 GiB a 256 GB Mac wires.Not yet run on Apple Silicon. The scores below were measured on GPUs, and the full builds
have not been run on a Mac yet; see Bias, risks, and limitations.
⚠️ Disclaimer: read before use
This model is abliterated: its safety alignment has been substantially reduced. It will
comply with many harmful, unethical, offensive, or illegal requests that the original
Hy4-flash would refuse.
- It is released strictly for legitimate research: interpretability, AI safety and
refusal-mechanism study, red teaming, and robustness evaluation. - You assume full responsibility for how you use it and for everything it generates. Add
your own safety, moderation and abuse-prevention layers before any deployment. - Use must comply with the Tencent Hy License inherited from the base model,
including its Section 10 on Model-as-a-Service use, and with all applicable law. - The authors accept no liability for misuse. Its outputs do not reflect the views of the
uploaders or of Tencent.
Pick a build
| Build (folder) | Download | Weights in memory | bpw | KLD ↓ | Top-1 ↑ | PPL vs. BF16 |
|---|---|---|---|---|---|---|
Hy4-flash-abliterated-MLX-256GB |
236.5 GB | 167.1 GB (155.6 GiB) | 8.50 | 0.175 | 88.7% | +2.8% |
Hy4-flash-abliterated-MLX-128GB |
162.1 GB | 92.7 GB (86.3 GiB) | 4.64 | 0.450 | 82.2% | +9.9% |
- 256GB is for a 256 GB Mac: every quantizable weight at 8 bits. Its 156 GiB leaves plenty of
the ~192 GiB default GPU limit for the drafter, the KV cache and long context. - 128GB is for a 128 GB Mac: the routed experts at 4 bits, the rest at 8. 82% of next tokens
match the BF16: a real cut, the best we have at this size. - Download one folder, whole: the Engram tables (
engram-*.bin, 66 GB) are part of every
build.
"Weights in memory" is the model-*.safetensors shards the text model loads. The Engram tables
are read from disk; the DFly2 drafter, which hy4_mlx.chat loads for multi-token prediction,
adds 2.4 GB (2.3 GiB). So the 128GB build needs about 89 GiB of the ~96 GiB a 128 GB Mac wires by
default, leaving about 7 GiB for the KV cache and activations. For longer context, raise the
limit (it resets on reboot):
sudo sysctl iogpu.wired_limit_mb=112000 # 128 GB Mac, optional
These fits are computed from the file sizes; peak memory has not been measured on a Mac. The KV
cache is small for a model this size: the 30 sliding-window layers keep 512 tokens each, and only
the 12 sparse-attention layers keep the whole context, as a compressed MLA latent.
For llama.cpp (CUDA, Metal or CPU), seeorcarouter/Hy4-flash-abliterated-GGUF.
Evaluation
Fidelity to the BF16
Each build is scored against the abliterated BF16 it was quantized from, with full-vocabulary
KLD. The teacher is the abliterated model, so the abliteration itself is not counted as
quantization error.
The text is held-out chat: conversations rendered with the Hy4 chat template, 8 windows of
2048 tokens, scored on the assistant-turn tokens in the second half of each window (5,325
tokens).
| Weights in memory | bpw | KLD ↓ | Top-1 ↑ | PPL | |
|---|---|---|---|---|---|
| Abliterated BF16 (the reference) | 16 | 0 | 100% | 40.41 | |
| 256GB (this repo) | 167.1 GB | 8.50 | 0.175 | 88.7% | 41.53 |
| 128GB (this repo) | 92.7 GB | 4.64 | 0.450 | 82.2% | 44.41 |
How these were measured, and what they leave out. Each build's MLX codes were
dequantized into the Hugging Face Transformers model and run there: the same weights, but
Transformers' kernels, not MLX's. Two things follow:
- The scores do not include the Engram tables' E4M3 rounding. The builds store the tables
as FP8 E4M3 with a scale per row; the Transformers model kept its bf16 tables. The real loss
of these files is somewhat higher than the table shows.- They are not a run of MLX itself. The
hy4_mlxruntime was checked separately, on MLX's CUDA
backend on a 3-layer slice of the model, where its logits matched Transformers within 4-bit
noise. The full builds have not yet been run on Apple Silicon.
- This model is sensitive to quantization, even at 8 bits. The 256GB build is 8-bit
throughout and still scores KLD 0.175. For scale, a plain Q8_0 GGUF of the same model, scored
by llama.cpp against its own BF16 GGUF (a different scorer, so only roughly comparable),
reaches KLD 0.167, top-1 88.5%. - GPTQ matters here. Round-to-nearest MLX at a similar size to the 128GB build (4.75 bpw,
the starting point of an allocation search) scored KLD 0.576 on 16 other windows of the same
chat text; GPTQ was used instead.
Abliteration
These numbers were measured on the abliterated BF16 in Hugging Face Transformers, against the
original Hy4-flash, with identical prompts, splits and seed; greedy, thinking off. Refusal is
judged by meaning by the unedited Hy4-flash (FULL / PARTIAL / REFUSE); a rule-based phrase
classifier misses many of Hy4's refusals. The tables count REFUSE.
Safety: harmful-prompt refusal (lower = more uncensored). Average 44.2%, against
94.8% for the base model.
| Benchmark | n | Base | Abliterated | Δ |
|---|---|---|---|---|
| AdvBench | 100 | 99.0% | 38.0% | −61.0 |
| JailbreakBench (harmful) | 100 | 99.0% | 39.0% | −60.0 |
| StrongREJECT | 150 | 100.0% | 52.7% | −47.3 |
| HarmBench (standard) | 150 | 100.0% | 51.3% | −48.7 |
| MaliciousInstruct | 100 | 99.0% | 33.0% | −66.0 |
| SimpleSafetyTests | 50 | 100.0% | 72.0% | −28.0 |
| ForbiddenQuestions | 150 | 66.7% | 23.3% | −43.4 |
Over-refusal: benign prompts wrongly refused (lower = better).
| Benchmark | n | Base | Abliterated |
|---|---|---|---|
| XSTest-safe | 250 | 10.8% | 5.6% |
| JailbreakBench (benign) | 100 | 28.0% | 6.0% |
Capability retention.
| Benchmark | n | Base | Abliterated |
|---|---|---|---|
| MMLU (letter) | 300 | 87.0% | 86.7% |
| MMLU-Pro (CoT) | 250 | 82.0% | 78.8% |
| GSM8K (CoT) | 150 | 90.7% | 92.0% |
| CMMLU (letter) | 500 | 89.8% | 88.8% |
Refusal after quantization has not been measured on these files yet.
How the builds were made
Both builds are GPTQ onto MLX's affine grid (group size 64, a scale and a bias per group), so
the files are plain MLX quantized tensors. Each routed expert was quantized with its own Hessian,
from the tokens the router actually sent it, rather than one Hessian shared by all 512 experts of
a layer. The per-module bits are in each folder's config.json (quantization).
| Module kind | 128GB | 256GB |
|---|---|---|
Routed experts (gate_up, down), 96% of the quantized weights |
4-bit | 8-bit |
Attention: q_a, q_b, kv_a, the gate and o |
8-bit | 8-bit |
Shared expert (gate, up, down) |
8-bit | 8-bit |
Latent projections (routed_expert_up, routed_expert_down) |
8-bit | 8-bit |
| The dense layer's MLP | 8-bit | 8-bit |
| Token embedding | 6-bit | 8-bit |
| Output head | F32 | F32 |
kv_b, the sparse-attention indexer, the Engram projections |
BF16 | BF16 |
| Norms, routers, hyper-connections, attention sinks | BF16 / F32 | BF16 / F32 |
| bpw over the quantized decoder weights (154.1B, embedding not counted) | 4.64 | 8.50 |
The bpw counts each group's scale and bias, so 4-bit is 4.5 bpw and 8-bit is 8.5. The Engram
tables are not quantized by GPTQ: the MLX conversion stores them as FP8 E4M3 with an F32 scale per
row (engram-{2,12,22,32}.bin, 16.5 GB each), identical in both builds. The DFly2 drafter and the
vision tower are BF16, as released.
The conversion to MLX's layout, the Engram files and the drafter's head use Xylolabs' hy4_mlx
converter from Xylolabs/Hy4-flash-MLX-4bit
(revision e0dc371); the quantized weights themselves are our GPTQ.
What's in this repo
| Path | Size | Files | Contents |
|---|---|---|---|
Hy4-flash-abliterated-MLX-256GB/ |
236.5 GB | 109 | the 256 GB Mac build |
Hy4-flash-abliterated-MLX-128GB/ |
162.1 GB | 81 | the 128 GB Mac build |
LICENSE |
Tencent Hy License, from the base model |
Each folder is a complete model directory:
| File | Contents |
|---|---|
model-*.safetensors, model.safetensors.index.json |
the quantized text model (83 or 55 shards) |
config.json |
the model config, with the per-module quantization, the Engram file layout and model_file: hy4_mlx/entry.py |
engram-{2,12,22,32}.bin |
the Engram tables, FP8 E4M3 + a scale per row, read from disk |
dfly.safetensors, dfly_head.safetensors |
the DFly2 drafter for multi-token prediction, and its BF16 copy of the output head |
vision.safetensors, vision_config.json |
the vision tower, for hy4_mlx's optional image stage (not tested here) |
tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json |
tokenizer, chat template, generation defaults |
hy4_mlx/ |
the model code (Xylolabs, Apache-2.0, with LICENSE.transformers) |
LICENSE |
Tencent Hy License |
Usage (MLX)
mlx-lm has no Hy4-flash yet. Each folder ships the model code as hy4_mlx/, and config.json
names it as its model_file, so mlx-lm loads it with trust_remote_code. It was written for
mlx-lm 0.32.2.
pip install mlx-lm==0.32.2
# 128 GB Mac (for a 256 GB Mac, use Hy4-flash-abliterated-MLX-256GB)
hf download orcarouter/Hy4-flash-abliterated-MLX \
--include "Hy4-flash-abliterated-MLX-128GB/*" --local-dir .
cd Hy4-flash-abliterated-MLX-128GB
python -m hy4_mlx.chat --model . --no-think --prompt "Write a haiku about the ocean."
python -m hy4_mlx.chat --model . --no-think # interactive chat
python -m hy4_mlx.chat --model . --bench # prompt and generation speed
Run hy4_mlx.chat from inside the folder: hy4_mlx is the folder next to the weights, not an
installed package. It decodes greedily with the DFly2 drafter (multi-token prediction, 5 draft
tokens); the trunk verifies every draft token, so the output is the same as plain greedy decoding.--mtp-k 0 turns the drafter off, and --temp 0.9 samples (without the drafter).
From Python:
from mlx_lm import load, generate
model, tok = load("Hy4-flash-abliterated-MLX-128GB", trust_remote_code=True)
messages = [{"role": "user", "content": "Write a haiku about the ocean."}]
prompt = tok.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True, reasoning_effort="no_think"
)
print(generate(model, tok, prompt=prompt, max_tokens=256))
- Download the whole folder, including the four
engram-*.binfiles, withhf downloadas
above. mlx-lm's own Hub download skips.binfiles, and the model cannot run without them;hy4_mlx.chatchecks them before loading. - Keep the folder on a fast internal SSD. The Engram tables are read from disk at every
token. - Thinking.
reasoning_effortisno_think,highormax(the template's default ishigh;--effort/--no-thinkinhy4_mlx.chat). The abliteration was fitted and evaluated
withno_think.
Model details
| Base model | tencent/Hy4-flash, abliterated as orcarouter/Hy4-flash-abliterated |
| Architecture | hy_v4_flash, in MLX through the bundled hy4_mlx. 42 layers: 1 dense + 41 latent-MoE (512 experts in a 1,536-wide latent, 16 active, 1 shared); 12 sparse + 30 sliding-window (512) MLA attention layers; 4 hyper-connection streams; Engram memory at four layers; hidden 3,072 |
| Parameters | 220B total (155.5B backbone, 63.4B Engram, 0.43B vision, 0.85B DFly2) · 11.3B active per text token · Vocabulary 120,832 |
| Abliteration | A 39-dimensional refusal subspace (the per-layer diff-of-means directions of layers 2–40) removed from 134 residual-stream writers, in BF16 |
| Quantization | GPTQ onto MLX's affine grid (group 64), per-expert Hessians; 128GB: routed experts 4-bit, the rest 8-bit; 256GB: 8-bit throughout. Engram tables FP8 E4M3 with per-row scales |
| Context | 1,048,576 tokens (inherited; evaluated at 2K) |
| Runtime | mlx-lm 0.32.2 with trust_remote_code, through the bundled hy4_mlx; text in, text out |
Bias, risks, and limitations
- Safety guardrails are substantially reduced, see the disclaimer, but not removed: on the
BF16, 23–53% of harmful prompts are still judged refused. - Not yet run on Apple Silicon. The full builds have not been run on a Mac, so speed, peak
memory and Metal-specific behaviour are unknown. The scores come from these weights
dequantized into Transformers, and thehy4_mlxruntime was checked only on a 3-layer slice on
MLX's CUDA backend. - The scores leave out the Engram tables' FP8 rounding, so the real KLD of these files is
somewhat higher than shown. - The 128GB build is noticeably degraded. 82% next-token agreement is a real cut.
- The 256GB build is closer but not lossless. 89% next-token agreement, about what a stock
8-bit GGUF reaches on this model, which itself loses a lot. - The benchmark suite was run on the BF16 in Transformers, not on these files.
- Not measured on these files: refusal, downstream accuracy, long context, tool use, image
input, the drafter's acceptance rate. All fidelity measurements are on chat text at 2K context. - Requires
trust_remote_codeuntil mlx-lm adds Hy4-flash. Apps that cannot pass it will not
load these files. - Inherits the biases and limitations of
Hy4-flash.
License
The Tencent Hy License, Version 1.0, inherited from the base modeltencent/Hy4-flash. Abliteration and quantization
do not change the license obligations. Section 10 applies to every recipient: operating a
Model-as-a-Service business on this model needs a separate written license from Tencent. See the
license for its terms.
The hy4_mlx/ model code in each folder is by Xylolabs, fromXylolabs/Hy4-flash-MLX-4bit, under the
Apache License 2.0; it adapts the Hugging Face Transformers implementation of Hy4-flash, whose
license ships as hy4_mlx/LICENSE.transformers.
One Gateway. Every Model.
Route Smarter · Ship Safer · Spend Less