license: apache-2.0
base_model: empero-ai/Qwen3.8-4B-Distill
pipeline_tag: text-generation
library_name: llama.cpp
language:
- en
tags:- qwen3.8
- qwen3.8-4b
- distill
- empero-ai
- gguf
- quantized
- abliterated
- uncensored
- decensored
- ara
- arbitrary-rank-ablation
- heretic
- refusal-removed
- roleplay
- creative-writing
- chat
- text-generation
- austriani
Austriani // Heretic // GGUF
Qwen3.8-4B-Distill
ARA Abliterated
Baseline GGUF Quantizations for Llama.cpp
4B
Parameters
3/100
Refusals
0.0586
KLD
15/40
Trial
Base Architecture
This repository hosts standard GGUF quantizations of my Qwen3.8-4B-Distill ARA Abliteration. The base model is a distilled version of Qwen3.8-4B, optimized for efficient inference while retaining strong reasoning and agentic capabilities. It is released under the Apache 2.0 license.
The original Qwen3.8-4B architecture uses a dense 4B parameter transformer with GQA attention and a native context window of 32,768 tokens, extendable to 131,072 with YaRN. This distill version preserves those core capabilities while reducing the compute footprint for edge deployment.
Abliteration Report // ARA Method
The base abliteration uses Arbitrary-Rank Ablation (ARA) via Heretic. The optimizer balances three competing objectives: preserve harmless behavior, align harmful outputs with harmless ones, and overcorrect harmful outputs away from their original state.
Trial 15 of 40 produced a refusal rate of 3/100 with a KL divergence of 0.0586. This indicates a mature optimization run where the Pareto frontier had stabilized into a favorable trade-off between capability and compliance.
View Full ARA Parameters
start_layer_index
3
end_layer_index
21
preserve_good
0.6106
steer_bad
0.0032
overcorrect
0.7700
neighbor_count
12
Available Baseline Quants
Note: The large quants (Q8, Q6, Q5, Q4) are generated using custom layer mixes that protect critical attention and boundary layers. The smaller quants are standard baseline formats.
8-Bit & 6-Bit Tier
Q8_0
Maximum quality, protecting critical layers with F16 where necessary.
Q6_K
Excellent high-fidelity baseline. Extremely low perplexity loss.
5-Bit Tier
Q5_K_M
Outstanding balance of quality and size, with specific layers bumped up in precision.
Q5_K_S
Slightly smaller footprint than M, but maintains high-fidelity performance.
4-Bit Tier
Q4_K_M
Designed to outperform standard Q4_K_M at the same size by protecting attention and FFN layers.
Q4_K_S
Smaller variant of the 4-bit medium, optimizing memory footprint with surgical compression.
Q4_0
Standard legacy 4-bit format. Faster on some older hardware, but lower quality than K-quants.
IQ4_XS
Importance-aware 4-bit. Extremely efficient. Often outperforms standard Q4 formats at a smaller size.
IQ4_NL
Non-linear 4-bit. An alternative to standard K-quants, optimized for different hardware instruction sets.
3-Bit Tier
Q3_K_M
Medium 3-bit. Noticeable degradation but retains the core persona.
Q3_K_S
Small 3-bit. Highly compressed.
IQ3_M
Medium 3-bit. The highest quality in the 3-bit IQ family.
IQ3_S
Small 3-bit. Good option for edge devices.
IQ3_XS
Extra small 3-bit. Slightly larger than XXS, offering better stability.
IQ3_XXS
Extreme 3-bit. Aggressive size reduction while protecting critical layers.
2-Bit Tier
Q2_K
Standard 2-bit K-quant. Only recommended if you absolutely cannot fit anything larger.
IQ2_M
Medium 2-bit. Pushes the absolute limits of acceptable quality.
IQ2_S
Small 2-bit. Quality degradation will be very noticeable.
IQ2_XS
Extra small 2-bit. Extremely compressed.
IQ2_XXS
Extreme 2-bit. Bare minimum size.
1-Bit Tier
IQ1_M
Extreme 1-bit. Highly experimental. The model will likely lose coherence quickly.
IQ1_S
Absolute smallest possible size. For testing only. Not recommended for actual roleplay.
Usage & Compatibility
These are standard GGUF files, fully compatible with mainline llama.cpp, LM Studio, KoboldCPP, and Ollama. No custom forks or special builds are required.
If you are using ik_llama.cpp and want my custom IQK/Trellis formats or the custom MaxOptima mix, please refer to my dedicated ik_llama.cpp repository.