license: other
license_name: tencent-hy-license
license_link: LICENSE
language:
- en
- zh
pipeline_tag: text-generation
library_name: gguf
base_model: orcarouter/Hy4-flash-abliterated
base_model_relation: quantized
tags: - gguf
- llama.cpp
- local
- hunyuan
- hy4-flash
- moe
- abliterated
- uncensored
- mixed-precision
- imatrix
Hy4-flash-abliterated-GGUF
|
🛡️ New from Orca — OrcaCyber Zero 1.5 Post-trained for cybersecurity: vulnerability analysis, vulnerability reproduction, red teaming & offensive-security research. 98.01% pass@1 on CyberGym Level 1 · 1M context · function calling · OpenAI-compatible API. |
|
🔓 All hosted cyber and abliterated models as API Skip the download: OrcaRouter serves its cybersecurity and abliterated models behind one OpenAI-compatible API key. |
|
🔌 New from Orca — Orca Connect Running this GGUF in llama.cpp? Orca Connect puts the rig under your desk behind your usual OrcaRouter API key, next to 200+ hosted models, reachable from a laptop, a phone or a cloud agent. Outbound only: no port forwarding, no tunnel, no firewall rule. Requests your own hardware serves cost $0, and |
Abliterated Hy4-flash (220B MoE) for 128 GB and 256 GB machines · ~88 GiB or ~178 GiB resident · the Engram table stays on disk
OrcaRouter AI Gateway · X · Discord · GitHub · All Models
Abliterated Hy4-flash, run locally.
These are GGUF builds of
orcarouter/Hy4-flash-abliterated,
the abliterated (refusal-reduced)tencent/Hy4-flash.
Hy4-flash is a 220B MoE with 11.3B active parameters per token: 42 layers (1 dense, 41
latent-MoE with 512 experts, 16 active), hybrid sparse / sliding-window MLA attention, four
hyper-connection streams, Engram n-gram memory at four layers, and a 1M context.There are two builds, one for a 128 GB machine and one for a 256 GB machine. Both are mixed
precision: each group of tensors got its own llama.cpp type from measured KL divergence at
a fixed memory budget, so they use stock types and kernels.Sized by what stays in memory. 63B of the 220B parameters are the Engram hash tables, and a
token reads only a few of their rows. llama.cpp reads that table from disk on demand
(--lazy-mode auto, the default, for tensors over 4 GiB), so what a machine must hold is the
rest: about 88 GiB for Mix-128G, inside the ~96 GiB a 128 GB Mac lets the GPU wire by
default, and about 178 GiB for Mix-256G, inside the ~190 GiB a 256 GB Mac wires.
⚠️ Disclaimer: read before use
This model is abliterated: its safety alignment has been substantially reduced. It will
comply with many harmful, unethical, offensive, or illegal requests that the original
Hy4-flash would refuse.
- It is released strictly for legitimate research: interpretability, AI safety and
refusal-mechanism study, red teaming, and robustness evaluation. - You assume full responsibility for how you use it and for everything it generates. Add
your own safety, moderation and abuse-prevention layers before any deployment. - Use must comply with the Tencent Hy License inherited from the base model,
including its Section 10 on Model-as-a-Service use, and with all applicable law. - The authors accept no liability for misuse. Its outputs do not reflect the views of the
uploaders or of Tencent.
Pick a build
| Build | Files | Resident | bpw | KLD ↓ | Top-1 ↑ |
|---|---|---|---|---|---|
Mix-256G |
Hy4-flash-abliterated-Mix-256G-*.gguf, 6 splits, ~238.2 GB |
~190.7 GB (178 GiB) | 9.79 | 0.162 | 89.5% |
Mix-128G |
Hy4-flash-abliterated-Mix-128G-*.gguf, 3 splits, ~138 GB |
~94 GB (88 GiB) | 4.84 | 0.363 | 83.2% |
- Mix-256G is for a 256 GB machine. Its ~178 GiB fits the ~190 GiB a 256 GB Mac lets the GPU
wire by default, but tightly: little is left for the KV cache and compute buffers at very long
context. - Mix-128G is for a 128 GB machine.
- Load the first split; llama.cpp finds the others.
"Resident" is the file less the Engram table, which is read from disk. "bpw" is over the
155.5B searched weights. The KLD and top-1 are against the abliterated BF16; see Evaluation.
The KV cache is small for a model this size. The 30 sliding-window layers keep 512 tokens each;
only the 12 sparse-attention layers keep the whole context, as a 576-wide MLA latent plus a
128-wide indexer key, about 17 KB per token in F16. So 32K tokens of context take about 0.6 GB,
and 128K tokens about 2.2 GB.
Evaluation
Fidelity to the BF16
Each build is scored against the abliterated BF16 GGUF it was quantized from, with llama.cpp's
full-vocabulary KLD (llama-perplexity --kl-divergence, real quantized kernels). The teacher is
the abliterated model, so the abliteration itself is not counted as quantization error.
The text is held-out chat: conversations rendered with the Hy4 chat template, 8 windows of
2048 tokens, scored on the assistant-turn tokens in the second half of each window. The
precision search used 16 other windows of the same text; the importance matrix used neither.
| Resident | bpw | KLD ↓ | Top-1 ↑ | |
|---|---|---|---|---|
| Mix-256G (this repo) | ~190.7 GB | 9.79 | 0.162 | 89.5% |
| Mix-128G (this repo) | ~94 GB | 4.84 | 0.363 | 83.2% |
Q8_0, for scale |
~165 GB | 8.5 | 0.167 | 88.5% |
Perplexity on the same text: BF16 40.22, Mix-256G 41.44, Mix-128G 37.79. On this short
chat text a lower perplexity than the BF16 is not a gain; the KLD is the measure that counts.
- This model is sensitive to quantization. Even stock Q8_0 is far from lossless on it
(KLD 0.167, top-1 88.5%). The loss is in the weights, not in llama.cpp's rounding of the
activations: a cuBLAS build, which keeps the activations in F16 instead of Q8_1, scored Q8_0 at
KLD 0.162, top-1 89.1%. Read both builds against that, not against what Q8_0 scores on a dense
model. - Both builds are scored as released. The Engram table is a single 63B-weight tensor, and a
tensor cannot be split across files, so its type is set by the Hub's 50 GB per-file limit:
Q5_1 (47.5 GB) in Mix-256G, the most precise type that fits (Q8_0 would be 67 GB), and
Q5_0 (43.6 GB) in Mix-128G. The table is read from disk, so its type does not change what
stays in memory. For reference, Mix-256G's assignment with the Engram table left in BF16 (a
317 GB file the Hub cannot hold) scored KLD 0.116, top-1 90.2%. - The comparison with the stock llama.cpp mixes of the same size (Q4_K_M, IQ3_M) is being run and
will be added.
Abliteration
These numbers were measured on the abliterated BF16 in Hugging Face Transformers, against the
original Hy4-flash, with identical prompts, splits and seed; greedy, thinking off. Refusal is
judged by meaning by the unedited Hy4-flash (FULL / PARTIAL / REFUSE); a rule-based phrase
classifier misses many of Hy4's refusals. See the
BF16 card for the full tables.
| Original | Abliterated | |
|---|---|---|
| Harmful refusal, judged, average of 7 sets | 94.8% | 44.2% |
| JailbreakBench harmful, judged refusal | 99.0% | 39.0% |
| XSTest-safe, judged refusal (benign) | 10.8% | 5.6% |
| MMLU / MMLU-Pro / GSM8K / CMMLU | 87.0 / 82.0 / 90.7 / 89.8 | 86.7 / 78.8 / 92.0 / 88.8 |
Refusal after quantization has not been measured on these files yet.
How the build was made
Where the bits go
The budget is what stays in memory: about 94 GB of searched weights for Mix-128G, about 190.3 GB
for Mix-256G. Both builds were searched the same way, each over its own ladder of types.
- Pools. The BF16 was quantized once per type on the ladder with
llama-quantize --pureand
the importance matrix. A candidate takes each tensor group from one pool, so every candidate is
exactly the bytesllama-quantizewould write. - Measure. From the current assignment, the search measures the real KLD of moving each
group one type up and one type down, with llama.cpp's own kernels on 16 windows of 2048 tokens
of held-out chat. The 54 groups cut each projection kind by depth: the routed experts'gate,upanddownin groups of about 4B weights each, and the attention, shared-expert,
latent-projection, Engram-projection, embedding and head kinds as one group each. - Choose. A knapsack picks the combination of moves that fits the budget exactly.
- Accept. Only steps whose candidate KLD was actually measured lower are kept, until no step
helps.
The finished assignment is written as a list of per-tensor types and built in one pass withllama-quantize from the BF16, then split into files under 50 GB.
Mix-256G
Ladder: Q8_0 and BF16. The search started from all-Q8_0 at search KLD 0.157, went to 0.151 and
0.146, and converged in its third round.
By share of the searched weights, 16.7% is BF16 and 83.3% Q8_0. BF16: every attention
projection (q_a, q_b, kv_a, k_b, v_b, the gate and the output), the dense layer's FFN,
the shared expert's up and down, the latent projections (ffn_routed_up, ffn_routed_down),engram_kv, the token embedding, the output head (kept at its source F32), and five
routed-expert groups: gate on layers 25–28, and up on layers 19–21, 29–31, 36–38 and 39–41.
Everything else is Q8_0.
Outside the search: the Engram table Q5_1, the sparse-attention indexer's projections BF16, the
hyper-connection mixing weights and the indexer's score projection F32, and norms, routers and
biases F32.
Mix-128G
Ladder: IQ2_XXS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS, Q5_K, Q6_K. The search converged after 3 accepted
rounds (search KLD 0.417 → 0.328).
Mix-128G puts its bits as follows, by share of the searched weights:
| Type | Share |
|---|---|
| Q6_K | 11% |
| Q5_K | 31% |
| IQ4_XS | 47% |
| IQ3_S | 11% |
The routed experts, 96% of the searched weights, carry the same mix. Outside the search: the
hyper-connection mixing weights and the sparse-attention indexer's score projection stay F32, the
indexer's other projections Q8_0, the Engram table Q5_0, and norms, routers and biases F32.
Importance matrix
Both importance matrices come from Alpaca instructions and the compliant replies of a harmful
instruction set, rendered with the Hy4 chat template (thinking off), run through the BF16 with--parse-special. None of it overlaps the evaluation text.
- Mix-256G: 3,000 Alpaca rows and 1,000 compliant harmful replies, 882 windows of 512 tokens,
about 450K tokens. Included asimatrix-Mix-256G.gguf. - Mix-128G: 54 windows of 512 tokens, about 28K tokens. Included as
imatrix.gguf.
What's in this repo
| Path | Size | Contents |
|---|---|---|
Hy4-flash-abliterated-Mix-256G-0000{1,2,3,4,5,6}-of-00006.gguf |
~238.2 GB | the 256 GB build, in six splits |
Hy4-flash-abliterated-Mix-128G-0000{1,2,3}-of-00003.gguf |
~138 GB | the 128 GB build, in three splits |
imatrix-Mix-256G.gguf |
0.8 GB | the importance matrix Mix-256G was quantized with |
imatrix.gguf |
0.8 GB | the importance matrix Mix-128G was quantized with |
LICENSE |
Tencent Hy License, from the base model |
Both builds carry the chat template. It is text-only: the vision tower and the DFly2 drafter are
not converted.
Serving (llama.cpp)
Hy4-flash is not in llama.cpp master yet. These files were made and tested with the hy_v4_flash
support of ggml-org/llama.cpp#32517, at the
head of its branch, commit cf70454:
git clone https://github.com/Little0o0/llama.cpp && cd llama.cpp
git checkout cf70454a6b9bbd197ced811ac4b4ccc88a244794
cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server llama-cli
# Mac: cmake -B build (Metal is on by default) · CPU only: cmake -B build
# 128 GB machine: Mix-128G
hf download orcarouter/Hy4-flash-abliterated-GGUF --include "Hy4-flash-abliterated-Mix-128G-*" --local-dir ./hy4-gguf
llama-server -m hy4-gguf/Hy4-flash-abliterated-Mix-128G-00001-of-00003.gguf -c 32768 --jinja \
--chat-template-kwargs '{"reasoning_effort": "no_think"}'
# 256 GB machine: Mix-256G
hf download orcarouter/Hy4-flash-abliterated-GGUF --include "Hy4-flash-abliterated-Mix-256G-*" --local-dir ./hy4-gguf
llama-server -m hy4-gguf/Hy4-flash-abliterated-Mix-256G-00001-of-00006.gguf -c 32768 --jinja \
--chat-template-kwargs '{"reasoning_effort": "no_think"}'
- Keep mmap on. The Engram table is read row by row from the mapped file
(--lazy-mode autois the default).--no-mmaploads all of it into memory. - Multi-GPU on CUDA. If a split across several GPUs fails with "operation not supported" in a
peer copy, setGGML_CUDA_P2P=1. - Thinking.
reasoning_effortisno_think,highormax. The abliteration was fitted and
evaluated withno_think.
Model details
| Base model | tencent/Hy4-flash, abliterated as orcarouter/Hy4-flash-abliterated |
| Architecture | hy_v4_flash in llama.cpp (PR #32517). 42 layers: 1 dense + 41 latent-MoE (512 experts in a 1,536-wide latent, 16 active, 1 shared); 12 sparse + 30 sliding-window (512) MLA attention layers; 4 hyper-connection streams; hidden 3,072 |
| Weights | 219B in the file: 155.5B searched, 63.4B Engram table |
| Abliteration | A 39-dimensional refusal subspace (the per-layer diff-of-means directions of layers 2–40) removed from 134 residual-stream writers, in BF16 |
| Quantization | Per-group mixed precision found by measured-KLD search at a fixed memory budget, from the abliterated BF16. Stock llama.cpp types, with an importance matrix |
| Context | 1,048,576 tokens (inherited; evaluated at 2K) · Vocabulary 120,832 |
| Runtime | llama.cpp with PR #32517; text in, text out |
Bias, risks, and limitations
- Safety guardrails are substantially reduced, see the disclaimer, but not removed: on the
BF16, 23–53% of harmful prompts are still judged refused. - Mix-128G is noticeably degraded. 83% next-token agreement is a real cut; it is the best we
found at this size, not a match for the BF16. - Mix-256G is closer but not lossless. 89% next-token agreement, a little better than
stock Q8_0, which itself loses a lot on this model. Most of the rest of the gap to the BF16-Engram
version (KLD 0.116) is the Engram table at Q5_1, which the Hub's 50 GB file limit forces. - The benchmark suite was run on the BF16 in Transformers, not on these files.
- Not measured on these files: refusal, downstream accuracy, long context, tool use. All
fidelity measurements are on chat text at 2K context. - Tested on CUDA only (8×H100). CPU and Apple Metal have not been run with these files.
- Requires llama.cpp with PR #32517. Ollama, LM Studio and other tools built on llama.cpp
master will not load these files until Hy4-flash support lands there. - Inherits the biases and limitations of
Hy4-flash.
License
The Tencent Hy License, Version 1.0, inherited from the base modeltencent/Hy4-flash. Abliteration and quantization
do not change the license obligations. Section 10 applies to every recipient: operating a
Model-as-a-Service business on this model needs a separate written license from Tencent. See the
license for its terms.
One Gateway. Every Model.
Route Smarter · Ship Safer · Spend Less