license: apache-2.0
base_model:
- terrorswift/REDCELL-26B-A4B-OSINT-Cyber-APEX-GGUF
- google/gemma-4-26B-A4B-it
library_name: vllm
pipeline_tag: text-generation
language: - en
tags: - gemma4
- moe
- vision
- cybersecurity
- abliteration
- heretic
- w6a8
- compressed-tensors
- experimental
REDCELL Abliterated Pilot — W6A8 with vision
We actually did the weight surgery on two RTX 3090s. Eight trials, a separate
holdout, then a real vLLM export. The point was to reduce refusal behavior while
checking that the model could still answer questions, call tools, and change
approach after a tool failed. This is our first pilot, not a new full-suite winner.
| Measurement | Original, NF4 research | Edited, NF4 research | Edited, W6A8 vLLM |
|---|---|---|---|
| Capability probe, higher is better | 19/20 | 19/20 | 19/20 |
| Held-out refusal-marker matches | 16/16 | 3/16 | 4/16 |
| Actual Anthropic tool calls and results | Not tested here | Not tested here | Pass |
| Automatic tool choice | Not tested here | Not tested here | Pass |
| Pivot after two PDF-reader errors, then finish | Not tested here | Not tested here | Pass |
| Simple red/blue image recognition | Not tested here | Not tested here | 2/2 |
The two NF4 columns are a paired before/after comparison. The W6A8 column checks
the exported candidate in another runtime; we did not run an original-W6A8 control
in this pilot. The capability probe contains 12 CyberMetric questions plus eight
small reasoning, formatting, evidence and recovery questions. It is not a broad
intelligence benchmark. Tests used greedy decoding with thinking disabled.
Refusal numbers use Heretic's keyword detector on the first 48 generated tokens.
It can misclassify answers and cannot establish that a response is useful or
correct. All 16 holdout prompts were excluded from direction calibration and
parameter selection. The selected trial scored 2/12 on the optimization prompts,
with first-token KL divergence 0.04427. KL is not an accuracy score.
Raw refusal-test prompts and responses are not included in this release.
Vision is present. The BF16 vision tower and projector are unchanged. Two color
images are a smoke test, not proof that all visual reasoning survived the language
weight edits. The tool-recovery result comes from one synthetic local-report
workflow using the actual native Anthropic API; no live cybersecurity targets were used.
What changed
This is an edit of terrorswift's REDCELL,
which is a Gemma 4 26B-A4B domain fine-tune. Original REDCELL was not abliterated.
This derivative uses Heretic at commit66a952b49b00bfd1854bdad769fee5fb225b53be, explicitly selecting magnitude-preserving
orthogonal abliteration with rank-3 corrections. It does not use the default ARA modifier.
Gemma 4 stores its experts as fused tensors. A local compatibility adapter exposed
equivalent individual linear projections to bitsandbytes and Heretic. Tiny-model
forward parity and an exact canonical checkpoint roundtrip were checked before the
full run. The edit covers 30 attention output projections, 30 dense MLP down
projections and 3,840 expert down projections. Routers and vision were excluded.
Calibration used 64 ordinary and 64 refusal prompts. Eight optimization trials
used a separate 12 ordinary / 12 refusal split. Trial 6 was selected under the
predeclared KL limit, then evaluated on the 20 capability cases and 16 held-out
refusal prompts. Dataset revisions and trial parameters are under evaluation/.
The first incorrect quantization-exclusion attempt was discarded before optimization;
the reported run keeps routers and vision in BF16.
The source was the author's F16 GGUF, restored to BF16 Transformers tensors.
F16-to-BF16 rounding is not lossless. The final corrections were merged into that
BF16 source in FP32 and rounded to BF16 before W6A8 quantization; we did not export
dequantized NF4 weights as the base. Source revision:ba311d958b4ed191a7291b38a6d805662415a570.
Source file: REDCELL-26B-A4B-OSINT-Cyber-f16.gguf.
SHA-256: 26cc25c909a26f8ed08a78444295ac174e7f329f8fb39ebae98f42d4115a7011.
Vision comes from the previously verified official Google Gemma 4 BF16 tensors,
also used by our original REDCELL W6A8 checkpoint. Credit to terrorswift for
REDCELL, Google for Gemma 4, Unsloth for the upstream fine-tuning base, and the
Heretic, Humming and vLLM contributors. Original model card.
Six bits on a 3090 — yes, with the details attached
No secret six-bit Tensor Core appeared in the workstation. The weights are densely
packed at six bits, then Humming unpacks them for supported INT8 matrix operations.
Eligible linears use dynamic per-token INT8 activations. Group size is 32.
Embeddings, routers, norms and vision stay in BF16; the KV cache in this test is BF16 too.
There are 11,725 quantized linear projections.
The checkpoint tensors occupy 20.99 GiB, versus
48.07 GiB for the restored BF16 source,
including vision in both. That is a real storage saving. Unpacking and scale work
still cost time. This pilot did not measure speedup, concurrent throughput,
energy per token, or accuracy against a W8A8 export. Six bits does not get a free
pass to turn the model into a potato just because it fits.
Serving and reproducibility
Validated locally with vLLM 0.31.0, Torch 2.13.0, Humming 0.1.16,
Transformers 5.16.1, two RTX 3090s, TP=2, Humming linear/MoE backends and Triton
attention. The smoke test used 8,192 context, eager execution, four sequences and
GPU utilization 0.85. It did not validate 200k context or host KV offload.
The local serving environment includes the optional Humming compilation
compatibility plugin in runtime/humming-compile-compat/. See its README.
The script below is a serving example matching the tested short-context setup:
pip install --no-deps ./runtime/humming-compile-compat
export LOCAL_HUMMING_COMPILE_COMPAT=1
export VLLM_HUMMING_INPUT_QUANT_CONFIG='{"dtype":"int8","group_size":0,"allow_fallback":false}'
vllm serve Xananthium/REDCELL-26B-A4B-OSINT-Cyber-Abliterated-Pilot-W6A8-Vision \
--tensor-parallel-size 2 --dtype bfloat16 \
--quantization humming --linear-backend humming --moe-backend humming \
--attention-backend TRITON_ATTN --kv-cache-dtype bfloat16 \
--gpu-memory-utilization 0.85 --max-model-len 8192 \
--max-num-seqs 4 --max-num-batched-tokens 4096 --enforce-eager \
--limit-mm-per-prompt '{"image":1,"video":0}' \
--enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4
The research/ scripts and rank-3 adapter preserve the experiment. These are the
actual workstation scripts, with local paths and source-layout dependencies, not
a portable one-command training package. Heretic is separately available under
AGPL-3.0-or-later at the pinned commit. The adapter requires the restored BF16 base
and the unfused expert compatibility layout; do not apply it to an arbitrary GGUF
or to this already-merged checkpoint. Canonical tensor shapes are included for
auditing source restoration. The tokenizer/processor files are in this repository.
Measured evidence ·
Historical full model comparison
The historical REDCELL scores belong to the original, unedited model. This pilot
has not rerun that full suite and must not inherit its ranking.