license: apache-2.0
base_model: huihui-ai/Huihui-Qwen3.8-27B-abliterated
tags:
- gguf
- ternary
- 1.75-bit
- hybrid
- abliterated
- uncensored
- qwen
- apple-silicon
- metal
- codebook
- speculative-decoding
- dflash2
- llama.cpp
pipeline_tag: text-generation
language: - en
- zh
- ru
☕ Support Local AI Development
Help me get a Mac mini 24GB to optimize local AI models up to 80B parameters.
Donate via Ko-fi →
Every contribution helps move the project forward.
Huihui-Qwen3.8-27B Abliterated (Hybrid PTQ1_0-LUT + CoreQ4_1)
Optimized hybrid quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated engineered for Apple Silicon (Mac Mini M6 16 GB Unified Memory, Metal GPU).
Benchmark Comparison: Current Champion vs Baseline
| Metric | Original FP16 Baseline | Previous Hybrid v1 | Our Champion (CoreQ4_1) |
|---|---|---|---|
| Model Package Size | ~54.0 GiB | 6.86 GiB | 6.43 GiB |
| Average Precision | 16.0 bpw | 1.85 bpw | 1.75 bpw (Hybrid PTQ1_0 + CoreQ4_1) |
| Perplexity (PPL) | 3.65 | 3.9214 | 3.6742 ± 0.3901 (Native FP16 Parity!) |
| Speculative Speed (DFlash2) | N/A | 14.2 t/s | 20.87 tokens/sec |
| Metal GPU Working Set | > 54 GB (OOM) | 7.1 GB | 6.43 GiB (100% Resident) |
Evolution History: What Was Tried & What Worked
- Native Ternary PTQ1_0 (6.2 GiB): Showed extreme compression efficiency, but lost slight nuance in attention output projections.
- Hybrid LUT + Calibrated Codebooks (6.86 GiB): Restored output projection and sensitive intermediate blocks via calibrated lookup tables. PPL recovered to
3.9214. - CoreQ4_1 Splicing with DFlash2 (Current Champion, 6.43 GiB): Promoted the sensitive core blocks to
Q4_1while leaving passive weights inPTQ1_0. Paired with an asynchronous DFlash-2 draft model, achieving full FP16 perplexity parity (3.6742) and 20.87 t/s speculative generation on Apple Silicon.
🚀 Help Us Bring 80B AI Models to Affordable Hardware
Support the AI Homelab Project
I'm working on optimizing large language models to run locally on affordable hardware — without relying on expensive NVIDIA GPU servers.
My next goal is to get a Mac mini with 24GB of unified memory. This hardware upgrade will help me push local model optimization further and experiment with and optimize models of up to 80 billion parameters.
Why does this matter?
I want to explore how far we can push local AI through:
- Advanced quantization techniques
- Memory-efficient model architectures
- Apple Silicon and Metal optimization
- Faster inference with lower hardware requirements
- Making powerful AI models more accessible to everyone
🎯 Our Next Milestone: Mac mini 24GB
Every contribution will help fund the hardware needed for further experiments, development, and optimization.
I will share benchmarks, technical findings, optimization results, and open-source work with the community.
You can help make this happen:
☕ Support the project on Ko-fi
Even a small donation makes a difference. If you cannot donate, sharing this project also helps enormously.
Thank you for supporting independent AI research and making local AI more accessible! ❤️
Quick Start (llama.cpp / llama-server)
# Run server with DFlash speculative decoding
./llama-server \
-m Huihui-Qwen3.8-27B-Hybrid-CoreQ4_1.gguf \
-md Huihui-Qwen3.8-27B-DFlash2-ft5-asym-Q2_K-FC-FullRank.gguf \
--spec-type draft-dflash --spec-draft-n-max 3 --spec-draft-n-min 1 \
--host 127.0.0.1 --port 11434 -ngl 99 -c 32768