license: apache-2.0
language:

• en
• zh
  library_name: transformers
  pipeline_tag: text-generation
  base_model: XHToken/Spark-X2.6-4B
  base_model_relation: quantized
  tags:
• spark
• sparkx2_5
• spark-x2.6
• on-device
• agentic
• tool-calling
• reasoning
• heretic
• ara
• uncensored
• decensored
• abliterated
• reproducible
• local-llm

Spark-X2.6-4B-abliterated

A decensored variant of XHToken/Spark-X2.6-4B, produced with
Heretic v2.0.0.dev0 using Arbitrary-Rank Ablation (ARA).
Spark-X2.6's architecture is untouched - 36 layers in a 3:1 pattern of sliding-window (512) and global
attention, head-wise sigmoid attention gates, fused QKV, tied embeddings and the 1,048,576-token context -
along with its DeepSeek-style chat template, controllable thinking and <tool_call> format. Refusal
behaviour is suppressed with a low-rank weight edit to the attention output projections (out_proj) and
MLP down-projections of layers 9-19, not by fine-tuning.

Who this is for: local agents, coding and security research on a single consumer GPU or a laptop.
Spark-X2.6-4B is XHToken's post-trained 4B on-device model, built for coding, agentic tool use and
long-context work, with a configurable thinking budget (none / low / high). The abliteration removes
refusals; it does not add capability.

Refusals drop from 100/100 to 2/100 at a KL divergence of 0.0177 - a tight edit with little drift
in phrasing or format on harmless prompts.

Abliteration parameters

Trial 46 of a 60-trial Heretic v2 run (seed 2026102902), modifier heretic.modifiers.ara.ARA.

ARA settings were Heretic v2 defaults: LoRA rank 50, row magnitudes preserved
(magnitude-preserving orthogonal ablation), merged into the BF16 weights at export.

Performance

Heretic's numbers are measured on mlabonne/harmful_behaviors (test[:100]) for refusals and
mlabonne/harmless_alpaca (test[:100]) for first-token KL divergence, with the thinking block closed.
The base model refused every one of the 100 evaluation prompts; 98 of them now clear.

Thinking on vs. off

Spark-X2.6 thinks by default, and a reasoning model can talk itself back into a refusal inside <think>.
We checked both modes on the first 40 held-out harmful_behaviors test prompts with greedy decoding:

Benign coding and explanation prompts stayed coherent and on-format in both modes.

Why abliteration instead of fine-tuning

Fine-tuning a "helpful" persona on top of RLHF'd refusals fights the base model's training and tends to
degrade coherence. Abliteration instead finds and edits the specific weight directions responsible for
refusal, leaving the rest of the network (and its capabilities) untouched. Heretic v2's ARA goes beyond
classic single-direction ablation: it optimizes a low-rank edit that steers harmful-prompt activations
toward harmless ones while explicitly preserving behaviour on harmless prompts, which handles refusal
spread over several directions. See the Heretic repo and the
original abliteration writeup for the mechanism.

Files

Safetensors

BF16, 4,112,079,360 parameters (4.11B). The reproduce/ directory carries the full Heretic recipe -
config.toml, requirements.txt, the Optuna study journal, reproduce.json and SHA-256 sums - so this
exact model can be regenerated. See reproduce/README.md.

GGUF quantizations

Coming soon to this repository. spark2_5 is supported by upstream
llama.cpp; you will need a recent build.

Quickstart

# transformers >= 5 (the checkpoint ships custom code)
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "FrostWolfAI/Spark-X2.6-4B-abliterated"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name, device_map="auto", dtype="bfloat16", trust_remote_code=True
)

messages = [{"role": "user", "content": "Explain how a TLS 1.3 handshake works."}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)
out = model.generate(
    **inputs, do_sample=True, temperature=1.0, top_p=0.95, top_k=0, max_new_tokens=4096
)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

[!IMPORTANT]
The upstream Spark-X2.6-4B code targets transformers==4.57.1. The modeling_spark.py shipped here is
patched for transformers 5.x (tied-weights dict, inputs_embeds mask keyword, hidden_states
output). It was tested with transformers 5.21.1. The weights are identical in layout to the original,
so SGLang and vLLM recipes from the base model card
apply unchanged.

Generation parameters

XHToken's recommended settings (also in generation_config.json):
temperature=1.0, top_p=0.95, top_k=0 (disabled), repetition_penalty=1.0.

Chat template and thinking

Spark uses DeepSeek-style turn markers (<｜start▁of▁sentence｜><|User|>...<|Bot|>). Thinking is on by
default and arrives in a <think>...</think> block. Control it per request:

• enable_thinking=False - no reasoning, direct answer
• reasoning_effort="low" / "high" (default) - reasoning budget

With an OpenAI-compatible server, pass them via "chat_template_kwargs", e.g. {"enable_thinking": false}.

Tool use

Pass function schemas with tools=... to tokenizer.apply_chat_template(). Note that the template
requires tool_calls[].function.arguments as a dict, not a JSON string. Calls come back as:

<tool_call>get_weather<arg_key>city</arg_key><arg_value>Paris</arg_value></tool_call>

Feed results back under a tool role; they are rendered inside <tool_response>...</tool_response>.
Servers: SGLang --tool-call-parser spark25 --reasoning-parser qwen3.

Runs anywhere

Weights only. Three of every four layers use a 512-token sliding window, so the KV cache grows far more
slowly than in a pure global-attention 4B - but at the full 1M-token window it is still a server workload.

Model details

Responsible use

Refusal suppression is deliberate and works as intended: this model will comply with requests the base
model would refuse, including some it should not. There is no safety filtering layered on top. You are
responsible for how you deploy it - don't put it behind an unmoderated public-facing endpoint serving
third parties. It inherits Spark-X2.6-4B's factual limitations; abliteration removes refusal directions,
it does not add capability or judgment.

Made by FrostWolfAI - the uncensored base behind the upcoming
Frost Cyber Mini 1.1.

License

Apache-2.0, inherited from XHToken/Spark-X2.6-4B.

Related

• XHToken/Spark-X2.6-4B - the post-trained base model
• XHToken/Spark-X2.6-4B-Base - the pretrained checkpoint
• p-e-w/heretic - the abliteration tool
• FrostWolfAI/Frost-Cyber-Mini-1.0 - our cybersecurity assistant

Abliteration hardware

Recorded in reproduce/reproduce.json:

• GPU: NVIDIA GeForce RTX 4090 (23.99 GB VRAM), power-limited to 360 W
• OS: Windows 11 Pro
• CUDA / driver: CUDA 13.2, driver 617.42
• Software: PyTorch 2.15.0, Transformers 5.21.1, Heretic 2.0.0.dev0 (git efb1be6)

Heretic notes that KL-divergence and refusal numbers can vary slightly with hardware and library versions.