license: apache-2.0
base_model:

• 0xKitkat/Ornith-1.5-35B-A3B-Uncensored
  base_model_relation: quantized
  pipeline_tag: image-text-to-text
  library_name: gguf
  tags:
• gguf
• llama.cpp
• qwen3
• moe
• vision
• uncensored
• long-context
• yarn

Ornith 1.5 Cyclos 1M Uncensored — GGUF (Q4_K_M)

GGUF build of a LoRA fine-tune of
0xKitkat/Ornith-1.5-35B-A3B-Uncensored,
with a YaRN context extension to 1,048,576 and a companion vision projector.

All credit for the underlying model belongs upstream — see Credits.
This repository contributes only a fine-tune, a context-window change, and a quantization.

Files

Both GGUF files are required for image/video input. Most runtimes pair them automatically
from the mmproj- prefix when they sit in the same directory. The config.json is there for
vLLM's GGUF plugin — see Runtime support for why that still is not
enough to load this model in vLLM.

Changes from the base model

Stated per Apache-2.0 §4(b):

1. LoRA fine-tune (SFT), merged into the base weights.
   r=16, alpha=16, use_rslora=true (so scaling is alpha/sqrt(r) = 4.0).
   Targets attention and shared-expert projections plus, via PEFT target_parameters,
   the fused MoE expert tensors mlp.experts.gate_up_proj and mlp.experts.down_proj.
   Training focus: coding, market/trading analysis, content classification, and
   tool/function calling.
2. Context extended 262,144 -> 1,048,576 via YaRN (factor 4.0,
   original_context_length 262144), written into the GGUF metadata.
3. Quantized to Q4_K_M; vision projector exported separately at F16.

The vision tower was not fine-tuned — every adapter tensor lives under
language_model.*, and model.visual.* is bit-identical to the base after merging.
Image and video ability is inherited from upstream, neither improved nor degraded.

Honest limitations

• The 1M context is extrapolation, not training. The base is natively 262,144 with
  no RoPE scaling; YaRN extends it at inference time. Quality degrades as you go
  further beyond 262,144, and this has not been measured. Treat 1M as an upper bound,
  not a validated working length.
• No audio. This architecture has no audio encoder. <|audio_start|>-style tokens
  exist in the tokenizer but are vestigial shared-vocabulary entries with nothing
  behind them; runtimes report audio: false. Audio needs an external ASR front-end.
• No benchmarks are claimed. No evaluation suite was run against this build.
  Upstream benchmark numbers do not transfer to it.
• "Uncensored" is inherited from the base, which reduced refusal behaviour. It does
  not mean every request is answered, and it does not remove your responsibility for
  how the model is used.
• Q4_K_M is a lossy quantization; at 1M context, q4_0 KV cache compounds that. Small
  reasoning budgets measurably degraded arithmetic accuracy in local testing.

Runtime support (tested)

Tested locally on 2 x NVIDIA RTX 5060 Ti (16 GB each, 31.7 GB total), CUDA 13.1, 59 GB system RAM.

vLLM + GGUF does not work — and why

A config.json is included in this repo, because vllm_gguf_plugin reads the config from the
GGUF's own directory. That is necessary but not sufficient. With it in place the loader gets
past config parsing and fails at weight mapping:

RuntimeError: Failed to map GGUF parameters (550):
  ['model.layers.0.linear_attn.dt_bias', 'model.layers.0.linear_attn.A_log', ...]

Qwen3.5-MoE is a hybrid: 30 of its 40 layers are linear (SSM-style) attention, and the plugin's
generic weights adapter has no mapping for those GGUF tensor names. Fixing it needs a per-architecture
adapter in vllm_gguf_plugin/weights_adapter/, which does not exist for this model. Nothing in this
repository can work around it.

The config is flattened (model_type: qwen3_5_moe, architectures: ["Qwen3_5MoeForCausalLM"]) rather
than the nested multimodal form, because the plugin calls AutoModelForCausalLM.from_config and the
composite config raises AttributeError: 'Qwen3_5MoeConfig' object has no attribute 'vocab_size'.
It is kept in case a future plugin release adds the missing adapter.

vLLM + BF16 works

Verified end-to-end with coherent generation:

vllm serve <bf16-repo> \
  --tensor-parallel-size 2 --max-model-len 4096 \
  --gpu-memory-utilization 0.92 --cpu-offload-gb 22 --enforce-eager

BF16 weights are ~67 GB against 31.7 GB of VRAM, so roughly 44 GB is offloaded to system RAM. That
works but is offload-bound: about 1.8 tok/s on this machine, versus 19-57 tok/s for the Q4_K_M
GGUF under llama.cpp. On this hardware the GGUF is the practical choice; BF16 in vLLM is worth it only
with more VRAM, where it also unlocks the MTP head (Qwen3_5MoeMTP) for speculative decoding — a head
llama.cpp discards, which is what the harmless blk.40.* unused tensor warnings refer to.

Long context in vLLM/SGLang comes from config.json: rope_parameters with rope_type: "yarn",
factor: 4.0, original_max_position_embeddings: 262144. vLLM 0.27.1 reads the transformers-5
rope_parameters key. Raise --max-model-len toward 1048576 only with the KV memory to match.

The 1M context, in detail

context_length in this GGUF is 1,048,576, reached by YaRN scaling
(factor 4.0, original_context_length 262144) baked into the file metadata. No flags needed —
llama.cpp reads it and reports n_ctx_slot = 1048576 on load.

What it is not: trained. The base model is natively 262,144 with rope_type: default and no
scaling block. YaRN is an inference-time frequency interpolation, applied here after the fact.
Quality past 262,144 is unmeasured — no long-context evaluation was run. Treat 1M as a
mechanical upper bound, not a validated working length. If you need confidence rather than reach,
run at the native length instead:

llama-server -m ornith-1.5-cyclos-1M-uncensored-q4.gguf -c 262144   # native, no extrapolation

Why 1M is even feasible here. Qwen3.5-MoE is hybrid: only 10 of its 40 layers use full
attention (every 4th), the rest are linear/SSM layers whose state is constant-size rather than
growing per token. So the KV cache is roughly a quarter of what a dense 35B would need — about
20 KiB per token at f16.

Measured memory, on 2 x RTX 5060 Ti (31.7 GB VRAM total):

At full 1M with f16 KV the process needed 41.5 GiB RSS; with -ctk q4_0 -ctv q4_0 the same
context fits entirely in VRAM. Two practical consequences:

• Quantize the KV cache for anything near 1M. It is the difference between fitting and not.
• Use --fit on, not -ngl 99. Forcing full offload allocated weights and KV successfully and
  then failed on compute buffers (failed to allocate CUDA0 buffer of size 4182280960). --fit
  places layers to match available memory.

One accuracy note: q4_0 KV plus a small reasoning budget measurably degraded arithmetic in local
testing (a compounding-percentage question came back 5.71% instead of 5.678%). For numeric work,
keep --reasoning-budget -1 and consider q8_0 KV at a shorter context.

Audio via a Whisper front-end

This model cannot hear. It has no audio encoder — see Honest limitations.
What works is transcribing first and passing the text in, which composes fine with the native
image/video path (audio transcript and an image in the same request both land in the prompt).

For convenience this repo ships whisper-ggml-large-v3-turbo.bin — an unmodified copy of
OpenAI's Whisper large-v3-turbo as distributed by ggml-org. It is not fine-tuned for this model
and is not covered by this repo's Apache-2.0 chain; it carries Whisper's own license and provenance.
You can equally fetch it yourself:

cd whisper.cpp && bash models/download-ggml-model.sh large-v3-turbo

audio_chat.py (also included) wires the two together — it converts any ffmpeg-readable file to
16 kHz mono, transcribes with whisper.cpp, and sends the transcript to a running llama-server:

python audio_chat.py meeting.m4a "Summarise the action items."
python audio_chat.py clip.wav --transcribe-only

Point it at your paths by editing WHISPER / WMODEL at the top, or set ORNITH_URL for a
non-default server address.

Run whisper on CPU (-ng, the script's default) while the LLM is loaded. The model occupies
~29 GB of a 32 GB dual-GPU rig; whisper then cannot allocate and segfaults inside
whisper_model_load rather than reporting OOM, which is easy to misread as a corrupt model file.
On CPU it runs at roughly 1x realtime with 16 threads.

Transcription loses everything non-verbal — tone, emotion, speaker identity, music, background
sound. Fine for meetings and dictation; a real limitation for audio classification.

Built with soup-cli

This build was produced with soup-cli — the LoRA
merge, the YaRN context config, and the GGUF conversion/quantization all ran through it.

# how this GGUF was produced (after merging the adapter into the base)
soup export -m ./ornith-merged -f gguf -q q4_k_m \
  --llama-cpp /path/to/llama.cpp \
  -o ornith-1.5-cyclos-1M-uncensored-q4.gguf

The 1M YaRN block was generated by soup's own helper rather than hand-written, so the values match
what soup would produce for any long-context export:

from soup_cli.utils.long_context import get_rope_scaling_config
get_rope_scaling_config("yarn", 1_048_576, 262_144)
# {'type': 'yarn', 'factor': 4.0, 'original_max_position_embeddings': 262144}

soup can also drive inference against the BF16 weights (it has no llama.cpp/GGUF serving backend —
its backends are transformers, vllm, sglang, mii):

soup serve  -m <bf16-repo> --backend vllm --tp 2 --max-model-len 262144
soup infer  -m <bf16-repo> -i prompts.jsonl -o out.jsonl
soup chat   -m <bf16-repo>

Usage (llama.cpp)

llama-server \
  -m ornith-1.5-cyclos-1M-uncensored-q4.gguf \
  --mmproj mmproj-ornith-1.5-cyclos-1M-uncensored-f16.gguf --no-mmproj-offload \
  -c 1048576 -ctk q4_0 -ctv q4_0 -fa on --fit on \
  --jinja --reasoning on --reasoning-format deepseek

Notes learned while building this:

• Quantize the KV cache at long context. 1M with f16 KV needs roughly 21 GiB for KV
  alone; q4_0 brings it to about 6 GiB.
• Prefer --fit on over -ngl 99. Forcing full offload can allocate weights and KV
  successfully and then fail on compute buffers.
• This is a reasoning model and emits <think>...</think>. Pass
  --reasoning-format deepseek to get it in message.reasoning_content. If you send
  images or video with a small max_tokens and thinking enabled, content can come
  back empty because the budget was spent on reasoning — raise max_tokens or set
  chat_template_kwargs: {"enable_thinking": false}.
• Per-request reasoning_budget_tokens works; reasoning_effort is ignored by this
  chat template.
• A recent llama.cpp is required — the architecture is qwen35moe.

Upstream recommended sampling: temperature=0.6, top_p=0.95, top_k=20.

Credits and attribution

This is a derivative several steps down a chain. Credit belongs to, in order:

• 0xKitkat — Ornith-1.5-35B-A3B-Uncensored,
  the direct base of this build (Apache-2.0).
• Ornith Team — ornith-ai/Ornith-1.5-35B-A3B,
  the coding/agentic post-training, vision tower, 262K context and MTP head (card declares MIT).
• Qwen Team — Qwen/Qwen3.6-35B-A3B (Apache-2.0).
• wangzhang / Abliterix — Qwen3.6-35B-A3B-abliterated,
  the abliterated donor and documented method.
• ggml-org — llama.cpp, whisper.cpp, the GGUF format, and the conversion/quantization tooling.
• OpenAI — Whisper large-v3-turbo, redistributed here unmodified for the optional ASR
  front-end under its own license; it is not a component of this model.

Licensed under Apache-2.0, inherited from the base model. Upstream components carry
their own licenses as listed above.