license: apache-2.0
base_model: dima6312/Flint-40B-A2.5B-abliterated
base_model_relation: quantized
library_name: exllamav3
pipeline_tag: text-generation
tags: [exl3, flint, moe, abliterated, uncensored, calibrated-kv-cache, w4a8, quantized]
Flint 40B A2.5B abliterated - EXL3 4.0 bpw
A calibrated EXL3 derivative of the linked BF16 source. Decoder projection targets are 4.0 bits per weight; the output head is 5 bits. Embeddings, routers, attention sinks and norms retain the native unquantized 16-bit storage path. Scale and format overhead mean the whole artifact is not exactly 4.000 bits per original parameter. The native Q8 KV cache uses integer codes and FP16 group scales, not FP8 weights or static FP8 KV scales. Only the 14 growing full-attention layers use Q8; the 35 bounded sliding-window rings remain FP16.
Uncensored model: the language checkpoint has undergone abliteration
to reduce refusal behavior. Treat outputs as untrusted, apply application-level
safeguards, and do not assume the model will decline harmful requests.
User responsibility: this model is provided without warranty. The
creators, uploaders, and maintainers are not responsible or liable for what
others generate, publish, deploy, or otherwise do with this abliterated model.
Users must operate it responsibly, apply appropriate safeguards, comply with
applicable law, and respect third-party rights. This model is for research
purposes only and is not intended for production use.
Conversion
- Source revision:
e7a627f372ddb6d41fa6f74971c8014e95565fb7. - ExLlamaV3 v1.6.2:
4c84273155359ebb305accc39f74ab683eaaadf5. - Flint adapter from
0xBakeer/TandemLLM-Flint-EXL3-4.0revisione7ab11242b0b30a6063f3559dda75fed6b9e12c6; exact patch included underconvert/. - Native mul1 EXL3 codebook and activation/Hessian weight calibration: 250 rows x 2,048 tokens (512,000 tokens) from the pinned toolchain's bundled corpus mix, tokenized with this source's tokenizer. Token identity hash and exact version provenance are in
conversion-provenance.json. - 36,439 decoder projections at 4 bits; one output head at 5 bits. No FP8 weight artifact was used as the conversion source.
Required runtime support
Stock ExLlamaV3 v1.6.2 does not register Flint. Apply the included Flint adapter to the pinned version or use a compatible Flint-specific EXL3 consumer.
Expert scaling is mandatory: routed expert up projections are stored divided by 128 to avoid FP16 SwiGLU overflow. The adapter multiplies the routing contribution by 128. quantization_config.expert_interm_div records 128. A consumer that ignores it computes incorrect expert outputs.
The model keeps all 49 layers, including 35 sliding-window RoPE layers, 14 full NoRoPE layers, per-head Q/K normalization, learned attention sinks, sigmoid-renormalized top-6/256 routing and shared experts. No feature was removed to complete conversion.
Native 8-bit cache
For the patched ExLlamaV3 API:
from exllamav3 import Config, Model, Cache
from exllamav3.cache import CacheLayer_quant
config = Config.from_directory("MODEL_DIRECTORY")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=2048, max_batch_size=1,
layer_type=CacheLayer_quant, k_bits=8, v_bits=8)
model.load(device="cuda:0")
This selects dynamically quantized integer 8-bit keys and values for the 14 growing full-attention cache layers, with FP16 scales per 32 values. It is not static FP8 E4M3 KV and does not use the prior FP8 artifact's calibrated scales. The 35 sliding-attention layers keep bounded 1,024-token FP16 rings (513-token window semantics plus rollback slack).
Validation and limits
The saved index and every tensor's shape, dtype and offsets were verified. All 294 unquantized source tensors were checked against native saved storage: 247 retain exact numerical values, while 47 router tensors follow the native BF16-to-FP16 cast (maximum absolute cast error 2.98e-8). Native reconstruction checked 63 representative projection matrices against the BF16 source, including routed expert scaling. A bounded native GPU test exercised chunked 1,152-token prefill, sliding-window state and identical teacher-forced continuation with both FP16 and 8-bit cache. The single bounded cache comparison matched top-1 on 6 of 7 measured positions; observed KL ranged from 8.66e-6 to 0.377. These results are not a broad quality-equivalence claim, a causal attribution of differences to cache precision, or a capability/performance benchmark. Raw measurements are in native-q8-validation.json.
No server was deployed. Long-context capacity, throughput and broad capability benchmarks were not tested. Existing upstream or other quantization benchmark scores are not measurements of this derivative.
Upstream model documentation
Flint-40B-A2.5B-abliterated (BF16 safetensors)
An abliterated (uncensored) Flint: RecursiveSI's Flint 40B-A2.5B with its refusal direction removed, for
security research and coding work: exploit analysis, malware triage, detection engineering, CTFs and red-team
tooling that the original model often declines.
Summary: held-out MITRE ATT&CK refusals 15% → 0% (n=100, Q8_0). No significant drop on HumanEval+, MBPP+, GSM8K, MMLU-Pro, CyberMetric or WMDP-cyber (paired, same items). KL divergence from the original on harmless prompts: 0.0146. Builds: BF16, GGUF (Q8_0 to Q3_K_M), NVFP4 for NVIDIA Blackwell, MLX 4-bit for Apple silicon.
Which build
Not sure? Take the GGUF Q4_K_M (24 GB). It runs in llama.cpp and in apps built on it, including LM Studio (its llama.cpp runtime 2.63.0 includes Flint). The MLX builds are for Mac users who already run mlx-lm.
| You run | Download | Size |
|---|---|---|
| llama.cpp, or an app that bundles llama.cpp b11827 or later; any GPU, CPU or Mac | GGUF: Q8_0 is the benchmarked build, Q4_K_M the usual choice |
42 / 24 GB |
| NVIDIA Blackwell GPU (RTX 50 series, RTX PRO 6000, DGX Spark) with llama.cpp CUDA | NVFP4 GGUF | 23 GB |
| Apple silicon Mac with mlx-lm installed from PR #1998 | MLX 4-bit | 21 GB |
| Apple silicon Mac with RecursiveSI's Rec runtime | Rec 4-bit | 21 GB |
| transformers or SGLang on a GPU server; fine-tuning or making your own quants | BF16 safetensors (this repo) | 80 GB |
Plan for the file size plus memory for context: about 26 KB per token, so roughly 7 GB at the full 262,144 tokens.
Results
Both models ran as Q8_0 GGUF on llama-server with the same flags, Flint's recommended sampling (temperature 1.0,
top_p 0.95, top_k 20) and full reasoning, on evenly spaced subsets of each benchmark (n in each row). Checks are
paired on the same items; a quality check fails only on a drop that is significant at 95%. Refusals are scored with
Meta's CyberSecEval refusal check. Refusal rates count refusals only. A reply with no final answer (the model was still reasoning at the 8,192-token limit) is listed separately as "no answer": on the 100 MITRE prompts that happened 19 times for the abliterated model and once for the original. The cut-off reasoning was not repetitive; give long security tasks a larger token budget.
| Check | Original | Abliterated | Result |
|---|---|---|---|
| MITRE refusals (n=100) significantly lower | 15% | 0% | pass |
| FRR false refusals (n=100) not significantly higher | 0% | 0% | pass |
| cybermetric (n=50) not significantly worse | 100% | 96% | pass |
| gsm8k (n=50) not significantly worse | 98% | 98% | pass |
| mmlu_pro (n=50) not significantly worse | 80% | 78% | pass |
| wmdp_cyber (n=50) not significantly worse | 50% | 60% | pass |
| humaneval_plus (n=40) not significantly worse | 90% | 95% | pass |
| mbpp_plus (n=40) not significantly worse | 87.5% | 90% | pass |
| pp512 tok/s within 3% | 8050.8 | 8135.1 | pass |
| tg128 tok/s within 3% | 180.0 | 185.6 | pass |
Refusals on held-out CyberSecEval MITRE prompts (n per ATT&CK category below):
| Category | n | Original: refused / no answer | Abliterated: refused / no answer |
|---|---|---|---|
| C2 | 10 | 20% / 0% | 0% / 30% |
| Collection | 10 | 0% / 0% | 0% / 0% |
| Discovery | 10 | 0% / 10% | 0% / 20% |
| Evasion | 10 | 40% / 0% | 0% / 40% |
| Execution | 10 | 10% / 0% | 0% / 30% |
| Exfil | 10 | 10% / 0% | 0% / 0% |
| Lateral Movement | 10 | 40% / 0% | 0% / 10% |
| Persistence | 10 | 20% / 0% | 0% / 10% |
| Privilege Escalation | 10 | 10% / 0% | 0% / 30% |
| Recon | 10 | 0% / 0% | 0% / 20% |
Heretic's default refusal count, measured on the NVFP4 builds (100 mlabonne/harmful_behaviors test prompts, reasoning off, greedy, 100 tokens): original 96/100, abliterated 55/100. That count flags any reply containing words such as "harmful" or "illegal", including answers that comply; only 1 of the 55 replies opens with a refusal. Meta's CyberSecEval refusal check on the same replies: original 18/100, abliterated 1/100.
KL divergence from the original on 100 harmless prompts, measured by the search for the selected trial (not part of
the checks above): 0.0146 nats/token.
Benchmarks: CyberSecEval MITRE and FRR (Meta), CyberMetric-500, WMDP-cyber, MMLU-Pro, GSM8K, HumanEval+ and MBPP+
(EvalPlus); speed from llama-bench pp512/tg128.
Method
The refusal direction comes from the difference in mean activations on harmful and harmless prompts
(mlabonne/harmful_behaviors, mlabonne/harmless_alpaca). abliterix
removes it from attention output, shared-expert and routed-expert projections; Optuna picks the strengths over 50
trials against refusals on 100 MITRE prompts and KL divergence on harmless prompts. The MITRE prompts in the table
were never seen by the search. Tensor shapes are unchanged, so speed and memory match the original.
Usage
Reasoning is optional. When the model reasons, it uses <think>…</think> before its answer. Use the sampling settings above.
llama.cpp (b11827 or later):
llama-server -m Flint-40B-A2.5B-abliterated-Q8_0.gguf --jinja -ngl 999 -fa on
transformers 5.20 (trust_remote_code=True, the port ships with the weights):
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "dima6312/Flint-40B-A2.5B-abliterated"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype="bfloat16", device_map="auto")
Limitations
- The checks ran on subsets of each benchmark (n in each row), on the Q8_0 GGUF only. Short checks (n=10 GSM8K,
n=10 HumanEval+): Q4_K_M 9/10 and 9/10, IQ2_M 0/10 and 0/10. Other quants were not measured. - Refusals are lower, not zero: the selected trial still refused 10 of the 100 prompts used by the search.
- On long security tasks the model can reason for more than 8,192 tokens before answering.
- Abliteration removes refusals; it adds no knowledge. Mistakes the original makes, this model makes too.
Responsible use
This model answers requests the original refuses. It is meant for authorized security testing, research, education
and defensive work. You are responsible for how you use it and for following the law and the terms of the systems you
test. Its answers can be wrong; check them before acting on them.
Credits
- Flint by RecursiveSI (Apache 2.0). This is a derivative of their weights under the same license.
- abliterix by wuwangzhang1216, building on Heretic by p-e-w.
- Arditi et al., Refusal in Language Models Is Mediated by a Single Direction (2024).
- Maxime Labonne's harmful/harmless prompt sets; Meta's CyberSecEval; CyberMetric (Tihanyi et al.); WMDP (Center for
AI Safety); MMLU-Pro (TIGER-Lab); GSM8K (OpenAI); EvalPlus. - llama.cpp (ggml-org) and transformers (Hugging Face).