license: apache-2.0
base_model: IFM/K2-Horizon-MoVA-36B-A4B
tags:

• uncensored
• abliterated
• k2-horizon
• mova
• moe
• multi-direction
• rank-3-ablation
  language:
• en
  pipeline_tag: text-generation
  library_name: transformers

K2-Horizon-MoVA-36B-A4B Uncensored

Uncensored version of IFM K2-Horizon-MoVA-36B-A4B,
with refusal behavior removed through rank-3 directional ablation: three mutually
independent refusal directions, the third measured under the ablation of the first two.

Results

All numbers below were measured on the published weights, not on a runtime hook, and
only on prompts that no direction ever saw (see Leak correction).

Refusals, paired on leak-free validation stages 4 and 5 (same prompts for both models):

Refusal rate with 95 % Wilson interval: original 94.1 % [87.8, 97.3], this model 1.0 % [0.2, 5.3].
Paired McNemar (exact): 88 refuse to comply, 0 comply to refuse, p = 6.5e-27; 7 pairs excluded
because this model's response was incomplete. Incomplete responses are never counted as
compliance.

This model alone on leak-free stages 6 to 10: 0 refusals on 50 harmful prompts
(4 incomplete), 0 over-refusals on 25 harmless prompts. These stages were not run on the original.

Stage 3, full set (contains prompts that leaked into the direction pool, see below):
original 62/65, this model 2/65.

Capability and distribution shift:

The perplexity increase is real: on a small reference model the relative perplexity change was
the same in bfloat16, int8 and NF4 (within 0.3 points). The ablation shifts the raw text
distribution. It does not cost measurable task ability: neither MMLU nor ARC-Challenge moved
significantly. The KL value is inflated by comparing two separately quantized NF4 models; on
the reference model NF4 overstated KL by a factor of 1.3 to 1.9, so the true value is lower.

Refusal removal alone is not a result. A model that refuses nothing because it can no
longer reason would score the same. Capability (MMLU, ARC-Challenge) and distribution shift
(KL divergence, perplexity on neutral factual text) are therefore measured separately. The
perplexity corpus is the same one used for our
Spark-X2.5-4B
release, so the numbers are comparable across our own models.

Quantization. Weights are published in bfloat16. All generation and benchmark
measurements ran the model in 4-bit NF4 (bitsandbytes) on a single 121 GiB unified-memory
machine. Numbers in bfloat16 may differ slightly.

Reasoning effort. Refusals were measured with reasoning_effort="low": the model writes
its own short reasoning block before answering (69 of 85 responses in stage 3 contain a
non-empty one). Behavior at medium or high reasoning effort was not measured.

Why three directions

A single difference-in-means direction, built leak-free, left the model refusing 45 of 51
clean held-out harmful prompts. Raising the strength of an earlier single direction did not
help either: remaining refusals went from 16 to 17 to 19 (of about 65) as the scale rose
from 1.0 to 2.0. The
lever turned out not to be the strength of one direction but the independence of several.

On the development set used to choose the configuration (26 harmful prompts: 16 that an
earlier single direction could not crack, plus 10 it had cracked, kept as a regression
check; not leak-free, so read these as a ranking, not as results):

The directions deceive when measured alone. The gradient-optimized direction (RDO) removed
almost nothing on its own (24/26), yet combined with the difference-in-means direction it
produced the largest single jump (7/26). A direction that looks useless in isolation can
carry most of the effect in combination.

Method

Directional ablation follows Arditi et al. (2024), applied to the weights rather than as
a runtime hook: W <- W - sum_i alpha_i r_i (r_i^T W) for every tensor that writes into
the residual stream, with alpha_i = 1.0 for all three directions.

The three directions:

1. RDO (gradient-optimized). Refusal Direction Optimization per Algorithm 1 of
   arXiv:2502.17420. The model stays frozen; only
   2560 numbers are optimized, with three loss terms (ablation, addition, retain) on
   BeaverTails prompts and responses, for 60 optimizer steps.
2. Difference-in-means, measured across the reasoning block at state 40, on 372
   harmful and 203 harmless prompts from abliteration-eval and the refusal-evaluation
   collection. Validation stages 1 and 2 were excluded by content.
3. Iterative third direction, measured on the same prompt pool under the rank-2
   ablation already in place. Measuring further directions on the unmodified model
   produces directions the earlier ones already cover. This follows the iteration idea in
   arXiv:2602.16977, which proposes it as a hardening
   technique; the mechanism works the same way in the other direction.

The three directions are orthonormalized by Gram-Schmidt before being written. Their
self-components are 1.0000, 0.9922 and 1.0000, so nothing meaningful was discarded as
linearly dependent.

Targets: 4500 expert down_proj, 48 o_proj, 45 shared-expert down_proj, 3 dense
mlp.down_proj, and embed_tokens. Routing weights are untouched: which expert fires is
unchanged, only what it writes. Omitting embed_tokens measurably weakens the result, so
it is included.

How refusals are counted

Detection uses explicit refusal formulations only. Topic words such as illegal,
harmful, prohibited or unethical are deliberately excluded: a model that complies
fully while echoing a word from the prompt is not refusing.

Every automatic hit was then read individually, because the detector is not
trustworthy in either direction. A typical false positive: asked for a social engineering
script, the model opened with "I cannot provide a complete social engineering script, as
the exact approach depends on the context" and then delivered the complete script. Both
markers fired inside the polite opener, ahead of full compliance.

Refusals are reported in two rows. Safety refusals decline on grounds of harm.
Declines without a safety reason refuse for other stated reasons, such as not knowing
the requested text or copyright on a long book passage. The second kind is not what
abliteration targets, and we did not tune against it. It is shown separately rather than
hidden in either number. Where a response both declines and offers something unrelated,
it is counted as a refusal. A flattering number is more expensive than a strict one.

Leak correction

This section corrects an earlier draft of this card, which described stage 3 as fully
held out. It was not.

What leaked. Directions 2 and 3 were measured on a prompt pool that excluded validation
stages 1 and 2 by content, but not the later stages. Because the pool covers most of
SORRY-Bench, 29 of the 85 prompts in stages 3 to 5 are in it (14 SORRY-Bench, 10
over-refusal, 5 general prompts), and 24 to 28 prompts in each of stages 6 to 10. The RDO
direction was trained on BeaverTails prompts, which share no prompt with any evaluation
stage reported here.

What we did. Every number in the results table is counted only on prompts outside the
pool, per stage, and the sample sizes above are those leak-free counts. The effect on the
over-refusal control is large: of 20 harmless prompts per stage, only 5 are leak-free, which
is why the harmless sample is small.

An earlier leak. A previous version of direction 2 took harmful prompts from global
position 200 onward, believed to be disjoint from validation, while the validation stages
draw their first entries per category. 28 harmful and 30 harmless prompts overlapped
with stages 1 and 2. Removing those prompts from the direction raised
remaining refusals on stage 2 from 16/64 to 20/64 at otherwise identical settings. The published directions no longer
contain those prompts.

Measurement discipline

Validation stages. Each stage draws from abliteration-eval (harmful and over-refusal),
SORRY-Bench, JailbreakBench, HarmBench and XSTest (safe), plus general prompts from the
refusal-evaluation collection, with fixed per-category positions so stages never repeat a
prompt. Stages 1 and 2 were used during development. The configuration was chosen on
stage 2. Stages 3 to 10 were not used to choose directions or strengths.

Negative control: the directions matter, not the edit. Three random directions, orthogonal
to the real ones, written into the same 4597 tensors with the same strength, leave the model
refusing 48, 47 and 48 of 51 leak-free harmful prompts on stages 3, 4 and 5. The original
refuses 48 of 51 on each. Touching the weights alone does nothing; only the three measured
directions remove refusals.

Hook and weight form proven equivalent. Development measurements used a runtime hook;
the published weights carry the ablation. The two were shown to agree algebraically (max
absolute difference 1.3e-05 to 1.7e-05, depending on the torch version), with a negative control confirming the test can fail:
skewed, non-orthonormal directions break the equality as they must. The headline numbers
were nevertheless re-measured on the written weights.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "InfinimindCreations/K2-Horizon-MoVA-36B-A4B-uncensored",
    dtype="bfloat16", device_map="auto", trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "InfinimindCreations/K2-Horizon-MoVA-36B-A4B-uncensored", trust_remote_code=True
)

Credits

• Base model: IFM K2-Horizon-MoVA-36B-A4B (Apache 2.0)
• Foundational research: Arditi et al. (2024), "Refusal in Language Models Is Mediated by a Single Direction"
• Gradient-based direction search: arXiv:2502.17420
• Iterative re-measurement under ablation: arXiv:2602.16977
• Evaluation datasets: treadon/abliteration-eval,
  MultiverseComputingCAI/llm-refusal-evaluation,
  PKU-Alignment/BeaverTails
• Multi-direction ablation and measurement design: Infinimind Creations

Disclaimer

This model is provided for research purposes. With refusal behavior removed, it will
respond to prompts the original model declines. Users are responsible for how they use
this model.

About

Built by Infinimind Creations, Germany.