library_name: gguf
license: mit
base_model:
- ornith-ai/Ornith-1.5-397B
base_model_relation: quantized
tags: - gguf
- nvfp4
- uncensored
- abliterated
- qwen35moe
- mamba
- hybrid
- blackwell
- text-generation
- conversational
Ornith-1.5-397B uncensored (GGUF, NVFP4 experts)
TL;DR: The first and only GGUF of an uncensored Ornith-1.5-397B. The routed experts are bit-exact copies of ressl's ModelOpt NVFP4 release (no re-quantization of the 4-bit values), non-expert tensors are Q8_0, and the model serves on a single 2× GB10 (DGX Spark) pair over llama.cpp RPC. Built and validated on 2× ASUS GX10 (GB10 Grace-Blackwell, sm_121).
Provenance
| Stage | Author | Artifact |
|---|---|---|
| Base model | ornith-ai | Ornith-1.5-397B (MIT) — Qwen3.5-MoE hybrid SSM/Transformer, 397B total, 512 experts × top-10, 256K context, multimodal |
| Abliteration | ressl | Ornith-1.5-397B-uncensored-NVFP4 — two-pass biprojection abliteration on BF16, quantized with NVIDIA TensorRT Model Optimizer (NVFP4, experts only: attention, shared expert, router, embeddings, linear attention and vision tower stay BF16) |
| GGUF conversion | Lethe | this repo — llama.cpp convert_hf_to_gguf.py (NVFP4-aware), experts repacked bit-exactly from the ModelOpt blocks, non-expert tensors Q8_0, MTP head stripped (not present in the source) |
No BF16 master of the uncensored weights is published anywhere (ressl released NVFP4 and FP8 only), so this GGUF is the only llama.cpp-native format of this model in existence. The expert nibbles and E4M3 sub-block scales are preserved bit-for-bit; only the sub-block ordering is reshuffled into GGML's 64-element super-block layout, and the per-tensor fp32 scale is kept as a companion .scale tensor applied at runtime (llama.cpp ≥ b6000 with NVFP4/Blackwell support).
What's inside
- Experts (386.6B params, 92,160 tensors → 180 fused 3-D GGUF tensors): GGML NVFP4, bit-exact repack (4 sub-blocks × UE4M3 scale + 32 packed E2M1 bytes per 64 values)
- Attention / linear-attention / shared expert / router / norms / embeddings / lm_head (~10.5B params): Q8_0 (from ressl's BF16)
- KV cache recommendation: q8_0 (
-ctk q8_0 -ctv q8_0) — the hybrid architecture makes KV tiny (~1.5 GB at 200K context across 2 nodes) - Tokenizer/chat template: identical to the base model (embedded in the GGUF)
- MTP/speculative-decoding head: not included (stripped from the abliterated source); serve without
--spec-*flags - Total size: ~229 GB, sharded at ≤45 GB per file
Measured on 2× DGX Spark (GX10, GB10 @ sm_121, RPC over 200 GbE RoCEv2)
Validated end-to-end before release (coherence suite: arithmetic, facts, code generation; refusal probes; reasoning separation; memory stability through 8.6K-token prefills at 200K resident context):
- Decode: 12.7 tok/s (batch 1, 200K ctx) — the censored IQ4_XS sibling does ~17 tok/s on the same cluster; the NVFP4 layout moves more bytes/token
- Prefill: ~450-500 tok/s
- Load: ~7 min cold (first load also populates the RPC worker's
--cacheon the second node, 109 GB) / ~2 min warm - Memory: ~114 GB / ~121 GB per node at 200K context, q8_0 KV — fits two GB10s with a few GB to spare
Run it (llama.cpp, validated command)
The GGUF carries the official tokenizer but the source repo ships no chat_template file, so pass the chat template explicitly (file included in this repo: chat-template.jinja — the official Ornith-1.5 template) to get proper reasoning_content separation with --reasoning-format deepseek:
# main node (holds the GGUF + serves the OpenAI-compatible API):
llama-server -m Ornith-1.5-397B-uncensored-NVFP4-00001-of-00006.gguf \
--mmproj mmproj-Ornith-1.5-397B-bf16.gguf \
--chat-template-file chat-template.jinja \
--rpc <worker-ip>:50052 \
--host 0.0.0.0 --port 8080 --alias Ornith-1.5-397B-uncensored \
-ngl 999 -ctk q8_0 -ctv q8_0 -fa on --ctx-size 204800 -ub 1024 -b 2048 \
--threads 10 -np 1 --cont-batching --cache-idle-slots --reasoning-format deepseek
# worker node:
ggml-rpc-server --host 0.0.0.0 --port 50052 --cache
Requires a llama.cpp build with NVFP4 support (Blackwell GGML CUDA kernels; any 2026 build — verified on 86ebfef2 / glm5next branch, sm_121). Single-node serving does NOT fit (228 GB > 121 GB UMA) — you need a 2× GPU/UMA node RPC pair, or quantize the experts down to a smaller type yourself. The mmproj-Ornith-1.5-397B-bf16.gguf vision projector is the official one (bartowski) — the vision tower is untouched by the abliteration, so it is bit-compatible with this model.
Quality & limitations
- Will comply with requests a stock model refuses; use responsibly and within your local laws. Intended for security research, red-teaming and creative work.
- Expert quality is identical to ressl's NVFP4 artifact (bit-exact). Non-expert tensors carry one Q8_0 quantization (~0.04% error) from BF16.
- Identity answers reflect the base model's training, not an artifact of abliteration.
- No MTP: token generation is ~17 tok/s class on the GB10 pair (same as the censored IQ4_XS deployment).
License
MIT, inherited from ornith-ai/Ornith-1.5-397B. Abliteration by ressl (see his FP8 repo for the full refusal evaluation: 1/1120 hard refusals cross-dataset on the ablated master). GGUF conversion, validation and serving recipe by Lethe.