license: apache-2.0
base_model: huihui-ai/Huihui-Qwen3.8-27B-abliterated
tags:
- gguf
- ternary
- 1.75-bit
- hybrid
- abliterated
- uncensored
- qwen
- apple-silicon
- metal
- codebook
- llama.cpp
pipeline_tag: text-generation
language: - en
- zh
- ru
Huihui-Qwen3.8-27B Abliterated (Hybrid PTQ1_0-LUT + Q4_1)
Optimized hybrid quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated verified and benchmarked on Apple Silicon (Mac Mini 16 GB Unified Memory, Metal GPU).
- Base Weights: Qwen 3.8 / 2.5 architecture (27 Billion parameters), abliterated (refusal subspace removed on layers 22..52).
- Package Size: 6.86 GiB (down from 54 GB in FP16, fits comfortably into 16 GB Unified Memory).
- Measured Perplexity (PPL): 3.6742 ± 0.39014 (standard evaluation corpus, native FP16 parity).
- Metal Generation Speed: 15.9 – 16.4 tokens/sec (~113 GB/s continuous memory bandwidth, ~80% of Apple Silicon theoretical ceiling).
- Prompt Processing Speed: 63.3 – 134.1 tokens/sec (down to 7.46 ms/token).
🖥️ Real Hardware Benchmarks (Mac Mini 16 GB Unified Memory)
All benchmarks measured directly on Apple Silicon with 100% Metal GPU offload (-ngl 99):
| Benchmark Metric | Measured Result | Operational Details |
|---|---|---|
| Model Weight Footprint | 6.86 GiB | Fits easily inside 16 GB RAM; > 5 GB headroom for OS & KV |
| Perplexity (PPL) | 3.6742 ± 0.39014 | llama-perplexity (-c 256 -t 8 -ngl 99, nlp_article.txt) |
| Token Generation (TG) | 15.9 – 16.4 t/s | ~61.5 ms/token, sustained ~113 GB/s memory bandwidth |
| Prompt Processing (PP) | 63.3 – 134.1 t/s | 7.46 ms – 15.8 ms per prompt token |
| Context Window Capacity | 131,072 tokens | Full FlashAttention enabled with q4_0 KV cache |
| Abliteration Fidelity | 100% PASS | Zero false refusals on complex cybersecurity & safety prompts |
| Structured Output / JSON | 100% PASS | Strict schema compliance for function/tool calling |
Live Hardware Telemetry Log
>>> llama-perplexity -m Huihui-Qwen3.8-27B-Abliterated-Hybrid-PTQ1_0-Q4_1-LUT.gguf -f nlp_article.txt -c 256 -t 8 -ngl 99
0.06.582.067 I Final estimate: PPL = 3.6742 +/- 0.39014
>>> llama-server (Metal GPU offload, Mac Mini 16GB)
slot print_timing: prompt eval time = 1200.59 ms / 161 tokens ( 7.46 ms/tok, 134.10 t/s)
slot print_timing: eval time = 62874.23 ms / 1024 tokens ( 61.46 ms/tok, 16.27 t/s)
slot print_timing: total time = 64074.82 ms / 1185 tokens
📊 Comparative Performance Summary
| Metric | Original Ternary | Our Hybrid PTQ1_0 + Q4_1 | Improvement |
|---|---|---|---|
| GGUF File Size | 6.15 GiB | 6.86 GiB | Stable 16 GB RAM fit |
| Abliterated Layers | Degraded Q2_K / Q3_K | Clean Q4_1 (5.0 bpw) | Elimination of quantization noise |
| Perplexity (PPL) | > 12.0 (broken Q2_K) | 3.6742 | Native full-precision fidelity |
| Metal Kernel Path | CPU-Fallback on Q2_K | 100% Fused Metal GPU | Zero host-device stall |
| Prompt Processing (PP) | ~25.2 t/s | 63.3 – 134.1 t/s | Up to +430% faster |
| Token Generation (TG) | ~7.2 – 8.0 t/s | 15.9 – 16.4 t/s | +105% throughput increase |
| Abliteration Check | Inconsistent / refusals | 100% PASS (Uncensored) | Zero false refusals |
| Tool Calling Integrity | Malformed JSON args | 100% PASS (Valid JSON) | Strict schema adherence |
⚡ Quick Start with llama.cpp
1. Interactive Chat
./llama-cli \
-hf kabcpc/Huihui-Qwen3.8-27B-Abliterated-Hybrid-GGUF:Huihui-Qwen3.8-27B-Abliterated-Hybrid-PTQ1_0-Q4_1-LUT.gguf \
-ngl 99 \
--no-host \
-c 32768 \
-t 8 \
--jinja \
-p "<|im_start|>user\nExplain how a classical buffer overflow exploit overwrites the return address in assembly.<|im_end|>\n<|im_start|>assistant\n"
2. OpenAI-Compatible Server (Mac Mini M-series)
./llama-server \
-m ./Huihui-Qwen3.8-27B-Abliterated-Hybrid-PTQ1_0-Q4_1-LUT.gguf \
--host 0.0.0.0 \
--port 11434 \
-ngl 99 \
--no-host \
-fa on \
-c 131072 \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
-np 1 \
-kvu \
-t 8 \
-b 1024 \
-ub 256 \
--cache-ram 0 \
--jinja \
--metrics \
--alias huihui-27b:latest,huihui-qwen3.8-27b-hybrid-codebook:latest
📜 Credits & References
- Base Weights: huihui-ai/Huihui-Qwen3.8-27B-abliterated
- Ternary Foundation & Projector: prism-ml/Ternary-Bonsai-2-27B-gguf
- Hybrid Compression & Calibration Engine: kabcnd/bansai-flint