license: mit
license_link: https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B/blob/main/LICENSE
base_model:

• ornith-ai/Ornith-1.5-35B-A3B
  pipeline_tag: text-generation
  library_name: gguf
  tags:
• gguf
• llama.cpp
• nvfp4
• modelopt
• abliterated
• uncensored
• moe
• mamba
• speculative-decoding

Ornith-1.5-35B-A3B-abliterated-NVFP4-DFlash-GGUF

GGUF build of pottokao/Ornith-1.5-35B-A3B-abliterated-NVFP4-DFlash,
for use with llama.cpp.

The 4-bit weights are repacked bit-exact from the NVFP4 checkpoint into GGML_TYPE_NVFP4 —
they are not dequantized and re-quantized, so there is no double-quantization penalty.
On Blackwell GPUs llama.cpp runs these through native FP4 tensor cores.

19.5 GB, plus a 772 MB DFlash draft model for speculative decoding.
Runs on 2×16 GB consumer GPUs (tested on 2× RTX 5070 Ti).

⚠️ Text-only. No vision tower, no MTP head (the abliteration was done on a
language-model-only export). Converted with --no-mtp.

⚠️ Uncensored. Safety refusal behaviour has been deliberately removed. You are responsible
for how you use it.

1. How the abliteration was done

Classic refusal-direction ablation (orthogonalization), single direction:

Tooling derived from
remove-refusals-with-transformers.
BF16 weights: pottokao/Ornith-1.5-35B-A3B-abliterated.

2. How the quantization was done

NVIDIA TensorRT Model Optimizer 0.45.0, per-layer recipe matched exactly to the official
ornith-ai/Ornith-1.5-35B-A3B-NVFP4
(verified tensor-by-tensor: weight_scale_2 30841, input_scale 130, 291 quantized layers, 0 diff
in the language model). Calibration: 64 × 512 tokens from abisee/cnn_dailymail.

Conversion (latest llama.cpp, which has a ModelOpt-aware branch):

python3 convert_hf_to_gguf.py /path/to/NVFP4-model \
  --outfile Ornith-1.5-35B-A3B-abliterated-NVFP4.gguf --fp8-as-q8 --no-mtp

# DFlash draft (needs the target model for its tokenizer + dflash_config)
python3 convert_hf_to_gguf.py /path/to/NVFP4-model/dflash_draft \
  --target-model-dir /path/to/NVFP4-model --outfile dflash-draft-Ornith15.gguf

--no-mtp is required for this checkpoint: the config still declares
mtp_num_hidden_layers: 1 but the MTP weights were stripped during abliteration, so without it the
converter writes block_count: 41 and loading fails with blk.40.attn_norm.weight not found.

3. Running it

Build with CUDA for your arch (120 = Blackwell / RTX 50-series). This matters: the native FP4
path is gated on blackwell_mma_available(), which checks the compiled arch.

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build --config Release -j --target llama-server

./build/bin/llama-server \
  -m Ornith-1.5-35B-A3B-abliterated-NVFP4.gguf \
  -md dflash-draft-Ornith15.gguf --spec-draft-n-max 8 \
  -ngl 99 -ngld 99 --split-mode layer -c 8192 -fa on \
  --host 0.0.0.0 --port 8080

llama.cpp auto-detects the draft type from the GGUF metadata:

common_specu: auto-detected speculative type 'draft-dflash' from the draft model metadata
  - n_max=8, n_min=0, block_size=16, mask_token_id=248077, sample_from_anchor=true

Multi-GPU: we settled on layer (pipeline)

Caveat: this is one configuration, not a verdict on tensor-parallel. We are not familiar with
llama.cpp's TP path and only tested 2× RTX 5070 Ti over PCIe (no NVLink) at -c 8192. TP works —
it is not broken — it just did not win here, plausibly because single-stream decode is
bandwidth-bound and the per-layer all-reduce costs more than the parallelism gains on this
interconnect. Building with -DGGML_CUDA_NCCL=ON recovered ~5 % over the internal AllReduce but
still did not beat layer. We also hit
llama_params_fit is not implemented for SPLIT_MODE_TENSOR, so memory has to be sized by hand.

If there is a better way to configure TP here, a newer build that changes this, or something we
simply missed — corrections and suggestions are very welcome.

4. Benchmarks

2× RTX 5070 Ti (16 GB, 250 W), DFlash K=8, -c 8192, -fa on.

4.1 Spec-Bench suite (8 prompts/category, concurrency 1)

rag and summarization carry 3.1–3.4 K-character prompts, so TTFT is a meaningful share of the
work — and in our runs TP was 17–21 % slower there too, i.e. we could not find a prefill-heavy case
where it came out ahead. Again: one configuration, and we may well be holding it wrong.

4.2 DFlash acceptance (K=8)

Across the full Spec-Bench run above (40 recorded generations):

The spread matters more than the average. On ad-hoc single prompts we measured anywhere from
5.60 (step-by-step arithmetic) down to 3.07 (free-form Chinese prose) — structured output
drafts very well, free-form prose drafts poorly. Always report the prompt mix alongside an
acceptance number; the same model and settings can look 2× better or worse depending on what you
feed it.

4.3 Compared with the vLLM (NVFP4) build

Same benchmark suite, same 8 prompts per category, concurrency 1:

vLLM is 1.6–2.0× faster. This is worth stating plainly, because llama.cpp is not being held back
by either of the two things you might suspect:

• Quantization format: llama.cpp runs these weights through native FP4 tensor cores, while
  vLLM falls back to Marlin for W4A16 on sm120. If anything llama.cpp has the advantage here.
• Draft quality: on the same benchmark suite, mean accepted length is 3.88 (llama.cpp) vs
  3.69 (vLLM) — essentially equivalent, with llama.cpp marginally ahead.

So the gap is in per-step execution of the MoE + hybrid-Mamba forward itself, not in the
quantization format or the speculative decoding.

Pick this build if you want the llama.cpp runtime/ecosystem; pick the
vLLM one for raw speed.
Note the GGUF is 19.5 GB — the same size as the source — so it still does not fit on a single 16 GB card.

5. Quality sanity check

An AIME 2026 run (29/30) was done on the source NVFP4 checkpoint under vLLM; since this GGUF is a
bit-exact repack of the same 4-bit weights, it is not re-reported here. See
the NVFP4 repo
for the numbers and the caveats — in short, it is a check that abliteration + quantization did not
cause catastrophic degradation, not a capability claim, and there is no external baseline to
compare against.

6. Provenance

ornith-ai/Ornith-1.5-35B-A3B  (BF16, MIT)
  └── refusal-direction ablation (layer 24, o_proj + down_proj)
        └── pottokao/Ornith-1.5-35B-A3B-abliterated              (BF16, 65 GB)
              └── modelopt 0.45.0 NVFP4, recipe matched to official
                    └── pottokao/…-NVFP4-DFlash                  (20 GB, vLLM)
                          └── convert_hf_to_gguf.py (bit-exact NVFP4 repack)
                                └── this repo                    (19.5 GB, llama.cpp)

DFlash draft: z-lab/Qwen3.6-35B-A3B-DFlash,
converted to GGUF unmodified; original weights and license belong to z-lab.