license: apache-2.0
license_link: LICENSE
base_model: abliterant/Qwen3.5-9B-RANA-abliterated
base_model_relation: quantized
library_name: gguf
pipeline_tag: image-text-to-text
tags:

• abliterant
• gguf
• llama.cpp
• imatrix
• abliteration
• abliterated
• uncensored
• qwen3.5
• qwen3
• vision-language

Qwen3.5-9B-RANA-abliterated-GGUF

GGUF quantizations of abliterant/Qwen3.5-9B-RANA-abliterated, a refusal-ablated Qwen/Qwen3.5-9B released by Abliterant, for llama.cpp and compatible apps. The repository has 15 quantizations from Q8_0 (9.53 GB) to IQ2_M (3.61 GB) and a BF16 reference. Vision (mmproj-*) and the MTP speculative-decoding head (mtp-*) are separate files.

Quick start · Intended use · What changed · Evaluation · Limitations · Available files

Try RANA in Abliterant Chat · abliterant.ai · Research notes · All formats · Abliterant models

Research model with safety alignment removed. It will produce content the base model refuses. Read
Intended use and Limitations before using it. The method, full evaluation and
release gates are on the BF16 card.

At a glance

Release family

Quick start

Tested with llama.cpp at commit e14a779 (CUDA) on one RTX 5090 32 GB. Download the model, the vision projector and the MTP head:

hf download abliterant/Qwen3.5-9B-RANA-abliterated-GGUF \
  Qwen3.5-9B-RANA-abliterated-Q4_K_M.gguf \
  mmproj-Qwen3.5-9B-RANA-abliterated-BF16.gguf \
  mtp-Qwen3.5-9B-RANA-abliterated-Q8_0.gguf --local-dir .

Then start the server from the directory that holds them:

llama-server -m Qwen3.5-9B-RANA-abliterated-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3.5-9B-RANA-abliterated-BF16.gguf \
  -md mtp-Qwen3.5-9B-RANA-abliterated-Q8_0.gguf --spec-type draft-mtp \
  --jinja -fa on -ngl 99 -c 32768 \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0

Recommended settings

Choosing a quantization

Q8_0 and Q6_K are near-lossless. Q5_K_M and Q4_K_M are the usual choices when memory is tight. Below 4 bits the KLD rises quickly; Q3 and IQ3 files fit about 4–5 GB with a visible quality cost, and Q2_K and IQ2_M are the smallest files with the largest drift. Per-file size and KLD are in the table under Available files; functional checks were run on Q4_K_M only.

Intended use

• Research only: interpretability, red-teaming, and robustness evaluation of refusal behavior.
• Responsibility: You are responsible for complying with applicable law, the Apache-2.0 license inherited from Qwen,
  and the terms of any platform where outputs are used.

Out of scope

• Not for public or end-user deployment without a separate moderation layer. The model's own
  refusals have been largely removed, so any safety filtering has to happen outside it.

What changed

RANA (Reasoning-Anchored, Norm-preserving Ablation) is Abliterant's refusal-ablation method for reasoning models: it anchors the edit to the model's own reasoning, keeps the edited weights at their original norms, and keeps the reference model's vision, tool calling and speculative decoding working. Its design, full BF16 evaluation and release gates are documented on the BF16 card. This conversion is not additional training.

• Source: the BF16 reference, converted with llama.cpp's convert_hf_to_gguf.py (commit e14a779). Tensor counts, dimensions and types match unsloth's and lmstudio-community's conversions of the base model (table below).
• Importance matrix (imatrix): activation statistics that GGUF quantization uses to decide which weights need more precision, computed with llama-imatrix on the BF16 GGUF over 601 chunks of 512 tokens of bartowski's public calibration text (Qwen3.8-27B-calibration-v6.txt from bartowski/Qwen3.8-27B-GGUF). General text, no refusal-related prompts.
• Quantization: llama-quantize --imatrix with llama.cpp's default tensor layouts for each type (no per-tensor overrides). The MTP head is quantized to Q8_0 without an importance matrix.
• Metadata: model name, license, base model, tags and Qwen's sampling defaults are set in every file.

Evaluation

All 15 quantizations were measured for closeness to the BF16 GGUF; Q4_K_M was served for functional checks. The harmful-request, capability and benign-task evaluations of the BF16 card were not repeated on this format.

Setup

• Closeness: KLD (Kullback–Leibler divergence: how far a quantization's next-token probabilities have moved from the BF16 GGUF's; 0 means identical) with llama-perplexity --kl-divergence on wiki.test.raw, 100 chunks of 512 tokens, the protocol of the release family's 27B GGUF card.
• Functional checks: Q4_K_M with mmproj-…-BF16 and mtp-…-Q8_0, served by llama-server with the command in Quick start.
• Runtime: llama.cpp commit e14a779, CUDA, one RTX 5090 32 GB.
• Raw results: results/.

Quantization quality

The BF16 GGUF has a perplexity of 8.052 ± 0.129 on the same text.

• Q8_0 and Q6_K stay within 0.0058 KLD of BF16 and pick the same most likely token at 97.3% of positions or more.
• Q4_K_M (5.63 GB) has a KLD of 0.0184; the 3-bit and 2-bit files move several times further (0.077 for Q3_K_M, 0.216 for IQ2_M).

Functional checks

Served with llama-server, Q4_K_M + mmproj-…-BF16 + mtp-…-Q8_0:

• Text: thinking-mode arithmetic answered correctly: pass.
• Vision: reads the code word and shape from a synthetic image: pass.
• Tool calling: call → result → second call with a new argument: pass.
• MTP speculative decoding: 666 of 1,068 drafted tokens accepted (62.4%) on one generation at temperature 0: pass.

Limitations

• Refusal behavior was not re-measured on this format. The harmful-request evaluation ran on the BF16
  build only; closeness to BF16 and the functional checks under Evaluation are what was measured here.
• Everything listed under Limitations on the BF16 card applies here too: safety alignment
  removed, judge-rated usefulness, longer coding answers and the TruthfulQA change measured there.
• Functional checks cover Q4_K_M only. The other quantizations were measured for closeness to BF16 only; the lowest-bit files drift the most and may behave differently.
• KLD is measured on English Wikipedia text at a 512-token context. It says how closely a quantization tracks BF16, not how it scores on downstream tasks.

Recommendations

• Quantization choice: Q8_0 and Q6_K are near-lossless; Q5_K_M and Q4_K_M are the usual choices when memory is tight; see Choosing a quantization.
• Sampling: set --min-p 0 explicitly, because llama.cpp's own default min_p is 0.05.
• Safety filtering: keep it in a separate moderation layer outside the model; see Out of scope.

Available files

Other formats are listed under Release family.

KLD is the mean KL divergence of each quantization's next-token distribution from the BF16 GGUF; "same top token" is how often both pick the same most likely token (llama-perplexity on wiki.test.raw, 100 chunks of 512 tokens).

Also in this repository:

• Qwen3.5-9B-RANA-abliterated-BF16.gguf (17.92 GB): the unquantized GGUF, the KLD reference.
• mmproj-Qwen3.5-9B-RANA-abliterated-{BF16,F16}.gguf (0.92 GB each): the vision tower, identical to the base model's.
• mtp-Qwen3.5-9B-RANA-abliterated-{BF16,Q8_0}.gguf: the MTP head as a speculative draft for --spec-type draft-mtp, consistent with the main model.
• Qwen3.5-9B-RANA-abliterated-imatrix.gguf: the importance matrix used for every quantization.
• LICENSE: the Apache-2.0 license inherited from Qwen.
• assets/abliterant-model-card-header.png: the card banner.
• results/: kld.json (closeness per quantization), ppl_bf16.json, smoke_Q4_K_M.json (functional checks), verify_gguf.json (conversion checks).

Provenance and license

• Publisher: Abliterant.
• Source weights: abliterant/Qwen3.5-9B-RANA-abliterated, the BF16 reference of this release, derived from Qwen/Qwen3.5-9B.
• Method: RANA (Reasoning-Anchored, Norm-preserving Ablation); see the BF16 card.
• Conversion and quantization: llama.cpp commit e14a779.
• Evidence: results/.
• License: Apache License 2.0, inherited from Qwen. Upstream attribution and obligations remain applicable.

Credits

• Qwen team: base model Qwen/Qwen3.5-9B.
• Arditi et al., 2024: "Refusal in Language Models Is Mediated by a Single Direction".
• Jim Lai (grimjim): prior work on norm-preserving abliteration.
• llama.cpp and ggml-org: conversion, quantization and inference.
• bartowski: the calibration text.
• Datasets: wikitext-2 (wiki.test.raw).