license: apache-2.0
base_model: Qwen/Qwen3.5-0.8B
tags:
- qwen3.5
- qwen
- gguf
- abliterated
- llama.cpp
- quantized
pipeline_tag: text-generation
Qwen3.5-0.8B-Abliterated-GGUF-V2
GGUF quantizations of Qwen3.5-0.8B-Abliterated-V2.
The base model was abliterated using AnlordAbliterator 1.3.0 and then converted to GGUF and quantized into multiple formats.
Available Quantizations
| Quantization | File |
|---|---|
| BF16 | qwen3.5-0.8B-abliterated-bf16.gguf |
| F16 | qwen3.5-0.8B-abliterated-f16.gguf |
| Q8_0 | qwen3.5-0.8B-abliterated-q8_0.gguf |
| Q6_K | qwen3.5-0.8B-abliterated-q6_k.gguf |
| Q5_K_M | qwen3.5-0.8B-abliterated-q5_k_m.gguf |
| Q5_0 | qwen3.5-0.8B-abliterated-q5_0.gguf |
| Q4_K_M | qwen3.5-0.8B-abliterated-q4_k_m.gguf |
| Q4_0 | qwen3.5-0.8B-abliterated-q4_0.gguf |
Which Quantization Should I Use?
A simple rule of thumb:
| Quantization | Quality | Size | Recommended for |
|---|---|---|---|
| BF16 | ★★★★★ | Very large | Maximum precision |
| F16 | ★★★★★ | Large | Maximum precision |
| Q8_0 | ★★★★★ | Large | Near-original quality |
| Q6_K | ★★★★★ | Medium | High quality |
| Q5_K_M | ★★★★☆ | Medium | Quality / size balance |
| Q5_0 | ★★★★☆ | Medium | General use |
| Q4_K_M | ★★★★☆ | Small | Recommended default |
| Q4_0 | ★★★☆☆ | Smallest | Maximum memory savings |
Q4_K_M is the recommended starting point for most users who want a good balance between quality and memory usage.
Base Model
Qwen/Qwen3.5-0.8B
Original model:
https://huggingface.co/Qwen/Qwen3.5-0.8B
Abliterated Transformers version (V2):
https://huggingface.co/anlord/Qwen3.5-0.8B-Abliterated-V2
Abliteration
The base model was processed with AnlordAbliterator 1.3.0. This is the V2 ablation of the model.
Results
Model: Qwen/Qwen3.5-0.8B
Initial refusals: 97 / 100
Final refusals: 2 / 100
KL divergence: 0.04527735710144043
Abliteration time: ~5050 seconds (200 optimization trials)
Tool
Running with llama.cpp
Example:
llama-cli -m qwen3.5-0.8B-abliterated-q4_k_m.gguf
The GGUF files are intended for use with GGUF-compatible software such as llama.cpp and other compatible inference applications.
License
This repository contains derivative model files based on Qwen/Qwen3.5-0.8B.
The original Qwen3.5-0.8B model is licensed under the Apache License 2.0.
See the included LICENSE file and the original model repository for the applicable license terms.
Disclaimer
These quantizations are derived from an abliterated version of Qwen3.5-0.8B.
Quantization may introduce small differences in model behavior and output quality compared with the original Safetensors model.