license: apache-2.0
base_model: orcarouter/Qwen3.8-27B-Uncensored
base_model_relation: quantized
library_name: mlx
pipeline_tag: text-generation
language:

• zh
• en
  tags:
• mlx
• apple-silicon
• qwen
• qwen3.8
• qwen3_5
• 3-bit
• mixed-precision
• imatrix
• quantized
• abliterated
• uncensored

Qwen3.8-27B-Uncensored · MLX 3-bit (mixed precision)

An MLX build of orcarouter/Qwen3.8-27B-Uncensored for Apple Silicon, quantized to fit in under 10 GB.

It is a 3-bit-class quantization: instead of one bit width for every layer, an importance-weighted plan gives each linear layer 2, 3 or 4 bits under a fixed size budget. The result averages 2.9 bits per weight including scales. That is a little smaller than a uniform 3-bit model (3.25 bits per weight at group size 128, 3.5 at 64).

⚠️ Disclaimer

The source model has had its safety alignment removed by abliteration, and this quantization keeps that behaviour. It will comply with harmful, unethical or illegal requests that the original Qwen3.8-27B would refuse.

• It is meant for research: interpretability, refusal-mechanism and safety studies, red-teaming and robustness evaluation.
• You are responsible for how you use it and for everything it generates. Don't deploy it to end users without your own safety and moderation layers.
• Use must follow the Apache 2.0 license and the laws that apply to you. The uploader accepts no liability for misuse.

See the source model card for the abliteration method and its refusal benchmarks.

Usage

pip install -U mlx-lm

Command line:

mlx_lm.generate --model nyaaorick/Qwen3.8-27B-Uncensored-MLX-3bit \
  --prompt "用三句话介绍一下杭州。" --max-tokens 1024

Python:

from mlx_lm import load, generate

model, tokenizer = load("nyaaorick/Qwen3.8-27B-Uncensored-MLX-3bit")
messages = [{"role": "user", "content": "Explain why the sky is blue."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=1024, verbose=True))

OpenAI-compatible server:

mlx_lm.server --model nyaaorick/Qwen3.8-27B-Uncensored-MLX-3bit --port 8080

The model thinks before answering by default. To turn thinking off, pass enable_thinking=False to apply_chat_template. Sampling defaults come from generation_config.json (temperature 1.0, top-p 0.95, top-k 20).

Hardware: weights take ~9.2 GiB of unified memory. A 16 GB Mac can run it with short contexts and little else open; 24 GB or more is recommended for long contexts.

Quantization

Bit allocation

• embed_tokens is 3-bit, group size 64.
• lm_head and every linear layer in the first two and last two blocks are fixed at 4-bit.
• Norms, convolutions and other small tensors stay in BF16.

Method

1. Activation statistics. One forward pass over ~100k calibration tokens records the mean squared input activation of every linear layer (an importance matrix).
2. Error per tier. Each layer's weights are quantized at each tier. The reconstruction error is weighted by those activation statistics, so input channels that matter more count more.
3. Greedy allocation under a byte budget. Every layer starts at 2-bit. The script repeatedly applies the upgrade (2→3 or 3→4) that removes the most weighted error per extra byte, until the 9.8 GB budget is spent. The budget counts actual bytes, including scales, the embedding table and the BF16 tensors, so the final size is guaranteed.
4. Apply. mlx_lm's quantize_model is run with the per-layer plan.

Calibration data: about 70% Chinese and 30% English. The Chinese text is from Wikipedia (wikimedia/wikipedia, 20231101.zh) and the English from the mlx-lm calibration_v5 text, cut into ~1000-token chunks and shuffled.

Hardware used: one NVIDIA A100 80 GB with MLX's CUDA backend (mlx 0.32.3, mlx-lm 0.32.0). Planning took ~8 minutes (peak 72 GB of GPU memory) and quantizing about 1 minute.

Evaluation

Perplexity with mlx_lm.perplexity: 128 samples × 512 tokens from allenai/tulu-3-sft-mixture, seed 123, same settings for both models.

No downstream benchmarks have been run on this build. With about 70% of the weights at 2-bit, expect a noticeable drop compared with 4-bit builds on hard reasoning, maths and long-form tasks.

Reproduce

The scripts in quantization/ produce this repo from the BF16 source:

hf download orcarouter/Qwen3.8-27B-Uncensored --local-dir ./src
python quantization/build_calib.py   # needs ./zh.parquet (zh Wikipedia shard) and ./en.txt
python quantization/imatrix_plan.py  # writes ./plan.json
python quantization/apply_plan.py    # writes ./out

Limitations

• Text only. mlx_lm drops the vision tower and the MTP speculative-decoding head of the source model.
• Safety guardrails removed (see the disclaimer). It also inherits the biases and limitations of Qwen3.8-27B.
• The quantization was calibrated mostly on Chinese text, and the Chinese Wikipedia text is largely Traditional characters.

License and credits

Apache 2.0, inherited from Qwen/Qwen3.8-27B. The full text is in LICENSE.

• Base model: Qwen/Qwen3.8-27B by the Qwen team.
• Abliterated BF16 weights: orcarouter/Qwen3.8-27B-Uncensored.
• This repo only adds the MLX conversion and mixed-precision quantization. It is not affiliated with Qwen, Alibaba or OrcaRouter.