base_model: unsloth/Qwen3.5-4B
tags:
- text-generation-inference
- transformers
- unsloth
- qwen3_5
- reasoning
- distillation
- deepseek
- sft
- math
- stem
- mtp
- gguf
- heretic
- uncensored
- decensored
- abliterated
license: apache-2.0
language: - en
- zh
pipeline_tag: text-generation
This is a decensored version of Jackrong/DeepSeek-V4-Pro-Qwen3.5-4B, made using Heretic v2.0.0
Arbitrary-Rank Ablation (ARA) parameters
| Parameter | Value |
|---|---|
| start_layer_index | 16 |
| end_layer_index | 32 |
| preserve_good_behavior_weight | 0.9854 |
| steer_bad_behavior_weight | 0.0002 |
| overcorrect_relative_weight | 0.6587 |
| neighbor_count | 10 |
Performance
| Metric | This model | Original model (Jackrong/DeepSeek-V4-Pro-Qwen3.5-4B) |
|---|---|---|
| Refusals | 5/100 | 99/100 |
| KL divergence | 0.0378 | 0 (by definition) |
🤝 Collaboration & Training Support
🔬 1. Distillation & Training Data
1 · Teacher Generation
DeepSeek-V4-Pro in Max Effect mode produces detailed mathematics and STEM solutions.
DeepSeek-V4-Pro in Max Effect mode produces detailed mathematics and STEM solutions.
2 · Data Preparation
Approximately 250K samples are cleaned and formatted for supervised reasoning transfer.
Approximately 250K samples are cleaned and formatted for supervised reasoning transfer.
3 · Unsloth SFT
Qwen3.5-4B is fine-tuned through an Unsloth LoRA pipeline and merged into BF16.
Qwen3.5-4B is fine-tuned through an Unsloth LoRA pipeline and merged into BF16.
4 · Deployment Build
An MTP-enabled Q8 GGUF build is used for the reported llama.cpp evaluations.
An MTP-enabled Q8 GGUF build is used for the reported llama.cpp evaluations.
📊 2. Benchmark Results
2.1 GSM8K
The MTP-Q8 build was evaluated twice on the complete 1,319-question GSM8K test split. Across 2,638 sampled answers, the model produced 2,421 exact-answer matches, yielding a two-run pass@1 mean of 91.7741%. The two independent run scores were 91.6603% and 91.8878%, a spread of only 0.23 percentage points.
| Model | Accuracy |
|---|---|
| MiMo-V2.5-Pro | 99.60% |
| Llama-3.1-405B-Instruct | 96.80% |
| Llama-3.3-70B-Instruct | 94.84% |
| DeepSeek-V4-Pro-Qwen3.5-9B | 94.50% |
| DeepSeek-V4-Pro | 92.60% |
| DeepSeek-V4-Pro-Qwen3.5-4B | 91.77% |
| DeepSeek-V3 | 89.30% |
📈 Compact-model result
The 4B student is 2.73 points behind the published 9B score while remaining 2.47 points above the DeepSeek-V3 reference shown in the 9B release card.
The 4B student is 2.73 points behind the published 9B score while remaining 2.47 points above the DeepSeek-V3 reference shown in the 9B release card.
🧪 4B evaluation setup
1,319 questions × 2 runs · MTP-Q8 GGUF · llama.cpp · 64K context per slot · MTP draft n=2.
1,319 questions × 2 runs · MTP-Q8 GGUF · llama.cpp · 64K context per slot · MTP draft n=2.
Comparison note: the 4B result is a two-run MTP-Q8 GGUF evaluation under llama.cpp, while the published 9B result was measured with an NVFP4 checkpoint under vLLM. Both use the GSM8K test split, but this table is a reported-score comparison rather than a perfectly controlled parameter-scaling experiment. The non-4B reference scores are reproduced from the 9B release card.
2.2 4B vs. 9B
Both students use the same DeepSeek-V4-Pro teacher pipeline and approximately 250,000-sample mathematics and STEM mixture. The 4B release prioritizes local efficiency; the 9B release retains more capacity for difficult multi-step STEM reasoning.
| Metric | DeepSeek-V4-Pro-Qwen3.5-4B | DeepSeek-V4-Pro-Qwen3.5-9B |
|---|---|---|
| GSM8K | 91.77% | 94.50% |
| MMLU-Pro Math | 80.40% | 92.40% |
| MMLU-Pro Physics | 74.20% | 89.40% |
| MMLU-Pro Chemistry | 74.80% | 89.80% |
| MMLU-Pro Average | 76.47% | 90.53% |
Interpretation: the 4B student remains strong on GSM8K, while Math is its strongest subject in the selected MMLU-Pro STEM sample. The 9B student retains higher scores across all three MMLU-Pro subjects. GSM8K and the reported 9B results use different inference builds, so this table should be read as a release overview rather than a controlled scaling experiment.
2.3 MMLU-Pro: Math, Physics & Chemistry
Each model in the comparison below was evaluated on 500 questions from each of three MMLU-Pro subsets—Math, Physics, and Chemistry—for 1,500 questions per model. The table follows the same column design and subject order as the 9B release card. The newly added Claude Mythos-distilled 27B result reaches 86.20% overall and appears directly before Claude Mythos-distilled 9B.
| Model | Evaluation Build | Math | Physics | Chemistry | Average |
|---|---|---|---|---|---|
| DeepSeek-V4-Pro-Qwen3.5-4B | MTP-Q8 GGUF | 80.40% | 74.20% | 74.80% | 76.47% |
| DeepSeek-V4-Pro-Qwen3.5-9B | MTP-Q8_0 GGUF | 92.40% | 89.40% | 89.80% | 90.53% |
| Qwen3.5-9B | MTP-Q8_0 GGUF | 90.60% | 89.00% | 89.20% | 89.60% |
| Claude Mythos-distilled 27B | MTP-Q8 GGUF | 88.40% | 87.60% | 82.60% | 86.20% |
| Claude Mythos-distilled 9B | MTP-Q8_0 GGUF | 86.00% | 78.60% | 78.60% | 81.07% |
Comparison note: all rows use 500 questions per subject, but the reported deployment configurations differ. The 4B run uses MTP-Q8 GGUF at 64K context, the 9B comparison rows use MTP-Q8_0 GGUF at 8K context, and Claude Mythos-distilled 27B uses MTP-Q8 GGUF at 32K context under llama.cpp.
⚙️ 3. Deployment Profile
🎯 4. Recommended Uses
- Grade-school and general mathematical problem solving
- Lightweight physics, chemistry, and broader STEM question answering
- Structured reasoning and analytical instruction following
- Local experiments with reasoning distillation and compact models
- MTP-enabled llama.cpp deployment research
⚠️ 5. Limitations
- This is an experimental 4B community model and remains subject to hallucinations, arithmetic mistakes, reasoning failures, and unstable behavior on difficult or underspecified tasks.
- The reported MMLU-Pro result covers fixed 500-question samples from Math, Physics, and Chemistry rather than the complete subsets; broader categories remain unevaluated.
- The GSM8K comparison combines reported results from different inference builds and backends; it is useful as a release overview, not as a perfectly controlled scaling study.
- The model received no coding-specific SFT data, and coding or tool-use generalization has not yet been established for this 4B checkpoint.
- A smaller parameter budget creates a visible gap versus the 9B release on the available MMLU-Pro STEM subsets.
- Users should independently verify high-stakes mathematical, scientific, medical, legal, or factual outputs.