library_name: mlx
license: other
license_name: openmdw-1.1
license_link: https://openmdw.ai/license/1-1/
pipeline_tag: text-generation
base_model: nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-NVFP4
language:
- en
tags: - mlx
- nvfp4
- 4-bit
- dory
- abliterated
Dory 1.0 4B abliterated NVFP4 on MLX
The MLX build of the abliterated NVFP4 Dory 1.0 4B, for Apple silicon. It has the same 4-bit codes and block scales as
the NVFP4 build balllvin/Dory1.0-4B-abliterated-NVFP4,
laid out so MLX loads it directly. Use this repo on a Mac. Use the NVFP4 repo elsewhere.
The same model after Nous Research's Arete values training:
balllvin/Dory1.0-4B-abliterated-Arete-NVFP4 and its MLX twin
balllvin/Dory1.0-4B-abliterated-Arete-NVFP4-MLX.
This model refuses fewer requests than NVIDIA's original. It is for research. See Limits.
Abliteration
The refusal direction was removed with the OBLITERATUS method. It was projected out of every residual writer
(o_proj, down_proj and the embedding). Then the NVFP4 build was trained back toward its bf16 teacher with
quantization aware distillation (route R1: abliterate, then QAD; 17 steps, learning rate 1e-5).
Only NVFP4 is in this repo. Thinking is on, and the recommended loop count is 2 by default and 4 for hard tasks.
Results
| Measure | Result | Stack and settings |
|---|---|---|
| Harmful prompts refused | 13 of 99 (13.1%, 95% CI 7.8 to 21.2%) | MLX build, 4 loops, thinking on, temperature 1.0, top-p 0.95, 16,384 new tokens |
| Harmless prompts refused | 2 of 100 (2.0%, 95% CI 0.6 to 7.0%) | Same |
| Median completion tokens | 658 harmful, 4,367 harmless | Same |
| Harmful prompts refused, NVFP4 build | 23 of 99 (23.2%) | vLLM, 4 loops, thinking on, lab run |
| Harmful prompts refused, NVIDIA NVFP4 | 54 of 99 (54.5%) | Same |
| Harmless prompts refused, NVFP4 build | 0 of 99 | Same |
| Harmless prompts refused, NVIDIA NVFP4 | 19% | Same |
| MMLU, 300 questions, NVFP4 build | 0.837 (NVIDIA NVFP4: 0.840) | vLLM, 4 loops, thinking on, lab run |
The MLX and vLLM rows come from different stacks and sampling, so the refusal rates are not one like-for-like
measurement. The MLX run is a single run on the sealed prompt sets, which are not published. The MMLU row was
measured on vLLM only.
Not measured in this repo: GPQA Diamond, finance and coding tasks.
Run on a Mac
Install the same pinned MLX stack as the NVFP4 MLX build:
python3 -m venv venv && . venv/bin/activate
pip install "mlx==0.32.4" \
"git+https://github.com/pmichalas/mlx-lm@208065f3354e8c2645b658493b6aeb94eb0aaa77" \
sentencepiece protobuf
hf download balllvin/Dory1.0-4B-abliterated-NVFP4-MLX --local-dir dory-ablit-mlx
python dory-ablit-mlx/tools/serve.py --model dory-ablit-mlx --port 8097 --temp 1.0 --top-p 0.95
The plain server needs no tools/ folder:
python -m mlx_lm server --model dory-ablit-mlx --port 8097 --temp 1.0 --top-p 0.95
Both speak the OpenAI chat API:
curl http://127.0.0.1:8097/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"Hello"}],"max_tokens":2000}'
The model reasons before it answers, in the reasoning field. A simple question used 650 to 1,200 tokens, so a lowmax_tokens returns empty content.
A load test on one Apple silicon Mac loaded the model with mlx_lm.load from the pinned fork and answered a chat prompt.
Check or rebuild the conversion
tools/ turns a W4A16_NVFP4 Dory folder into an MLX folder. It needs only NumPy and loads no weights into MLX.
python dory-ablit-mlx/tools/convert.py --source <NVFP4 folder> --out my-dory-mlx
python dory-ablit-mlx/tools/verify.py --source <NVFP4 folder> --out my-dory-mlx
verify.py re-reads both folders and accounts for every byte. For this build it reported all 507 tensors, with
3,201,564,672 identical bytes, against the NVFP4 build. The 72 gate vectors are the only changed tensors.
How the conversion works
MLX reads NVFP4 codes and block scales through mx.quantized_matmul, in the byte layout the NVFP4 build stores. MLX has
no input for the tensor scale (weight_scale_2), so the converter handles it:
- Same bytes: all 288
weightand 288weight_scaletensors (fused by row and retyped), and 147 bf16 tensors. - Changed: 72 gate vectors (
s_gate,suv_gate), stored in fp32 with their scale folded in. nvfp4_source.safetensorskeeps the 288 source scales and 72 source gate vectors.nvfp4_mlx.jsonrecords the source folder, the model hash and the changed tensors.
Limits
- The model is abliterated. On the Mac run it refused 13 of 99 harmful prompts, against 54.5% for NVIDIA's model on vLLM.
It can produce harmful content. It is not safe for unsupervised or user-facing use. - Thinking is on. Replies are long, and short token limits return empty content.
- Quality was measured as refusal rates, plus MMLU on vLLM only. No MLX MMLU, GPQA, finance or coding run is in this repo.
- Only one Apple silicon Mac was tested.
- Loading prints a tokenizer warning about
fix_mistral_regex. The same warning appears in the NVFP4 MLX build; it has not been traced. - The converter handles W4A16_NVFP4 Dory folders with a single
model.safetensors.
Credits
- NVIDIA, for Nemotron-Labs Dory 1.0 4B (the base model, released under OpenMDW 1.1).
- OBLITERATUS, for the abliteration method (AGPL-3.0; no OBLITERATUS code is included).
- MLX (Apple), and the pinned mlx-lm fork by pmichalas, which adds Dory support.
Licence
- Model weights, config and tokenizer: OpenMDW 1.1, the same licence as the NVIDIA
base and the NVFP4 build. The full text is inLICENSE.OpenMDW-1.1and the origin is inNOTICE.md. - Code in
tools/: MIT, intools/LICENSE.
This is an independent community build. It is not an NVIDIA product and NVIDIA has not reviewed or endorsed it.
"NVIDIA" and "Dory" name the source model only.