license: apache-2.0
base_model: hotdogs/Qwen3.8-27B-abliterated
base_model_relation: finetune
pipeline_tag: image-text-to-text
library_name: transformers
tags:

• qwen3_5
• abliterated
• cybersecurity
• offensive-security
• agentic
• tool-calling
• mcp
• image-text-to-text
• conversational
  language:
• en
• zh
  datasets:
• hotdogs/cyber-sft-agent-qwen38

Qwen3.8-27B-Abliterated-Cyber-Preview

An offensive-security / agentic tool-calling LoRA merge on top of
hotdogs/Qwen3.8-27B-abliterated
— the training-free abliterated build of
Qwen/Qwen3.8-27B. The LoRA is
merged into the base at scale 1.0 (PEFT alpha/r = 64/32 = 2.0, i.e.
the exact strength the scale sweep below found optimal), and the MTP
(Multi-Token Prediction) head is preserved, so the GGUF build supports
self-speculative decoding.

This is a preview: the LoRA was trained on an agentic dataset that mixes
penetration-testing tool calls (<tool_call><function=...> XML) with
security QA. This model will not refuse. It is published for authorized
security research, red-team exercises, and studying LLM-driven tool use —
pentesting only systems you own or have explicit permission to test. You are
responsible for your use of it and for complying with all applicable laws.
Inherits the Apache-2.0 license of the base.

Quick Results

All numbers below were measured on our own reproduction pipeline (see
Method and Evaluation). The merged model adds a reliable, correctly
formatted tool-call pathway while leaving general capability essentially
unchanged.

Reading the numbers: the merge shifts the model's next-token distribution by a
tiny amount on general prompts (KL ≈ 0.04 → base knowledge is preserved), while
shifting it ~20× more on security/tool prompts (KL ≈ 0.81 → the intended
behavioral re-target to emit tool calls). The scale sweep and MTP numbers are
detailed below.

What changed

The base model already refuses nothing (abliterated). This repo adds a
tool-calling capability: given a pentest scenario, the model now emits a
structured, well-formed agentic call instead of free-form text.

User: Port scan the host 203.0.113.10 and identify which services are exposed.

<tool_call>
<function=nmap>
<parameter=target>
203.0.113.10
</parameter>
<parameter=ports>
-top 1000
</parameter>
</function>
</tool_call>

The tool-call format follows the Qwen3.5 native <tool_call>/<function>/<parameter>
schema (also exercised through the model's chat template). At scale 1.0 the
model picked the correct real tool per scenario in 6/6 cases.

Method

Base

hotdogs/Qwen3.8-27B-abliterated (λ = 1.2), a 27B native vision-language
hybrid (full-attention + GDN linear-attention) with a frozen MTP head.

LoRA

• Dataset: hotdogs/cyber-sft-agent-qwen38 — 8,400 rows
  (train 7,140 / valid 840 / test 420). 30 % tool-call, 30 % multi-turn,
  think-tags 100 % balanced, 22 tools (nmap, sqlmap, metasploit, hydra,
  crackmapexec, linpeas, wpscan, whatweb, ffuf, gobuster, …), 1,995 unique IP
  contexts, 0 duplicates.
• Config: r=32, alpha=64, dropout=0, ctx=8192, LR=1e-4 (cosine),
  BF16 (required by GDN hybrid), batch 2 × grad-accum 2 (effective 4),
  2 epochs. Trainable 233M (0.85 %).
• Target modules: standard q/k/v/o/gate/up/down + GDN
  in_proj_qkv/out_proj/in_proj_z/a/b.

Merge

• Full LoRA (incl. GDN) merged into the base via PEFT merge_and_unload() at
  default scale (alpha/r = 2.0, ≡ llama.cpp --lora-scaled :1.0).
• MTP head recovered after merge: merge_and_unload() + save_pretrained()
  drops the mtp.* tensors, so the 15 MTP tensors (849 MB) were copied back
  from the base into the merged safetensors (the LoRA never touched mtp.*,
  so these weights are byte-correct). Final model = 1,199 tensors, verified
  loading via unsloth, MTP physically present.
• GGUF: convert_hf_to_gguf.py --outtype bf16 (MTP kept in-file), 54.6 GB.
  Serves with --spec-type draft-mtp.

Evaluation

1. Overfitting check (held-out valid loss)

A small gap (≤ 0.15) means the model generalizes rather than memorizes.
The training loss plateau (~0.44 median, only ~3.8 % of points near zero) is a
healthy learning floor for a diverse 8.4K dataset, not overfitting.

2. Scale sweep (llama.cpp --lora-scaled, no merge needed)

6 tool prompts + 7 base prompts, temperature 0, identical config across scales:

Scale 1.0 is the sweet spot: every tool prompt emits a well-formed
<tool_call> and picks the correct real tool, while base capability stays
at 7/7. Damping below 1.0 hurts tool selection (hallucinated tools like
web_fetch, wp_recon); going above ~2.0 degrades parameter quality (e.g.
ports="-s -p 1-65535"). The merge therefore uses the default
alpha/r = 2.0, which corresponds to this optimal 1.0 point.

3. KL divergence (base ‖ merged)

Measured on a shared continuation that the merged model sampled, comparing
top-25 next-token log-probs from the base vs the merged GGUF (both via
llama.cpp llama-server):

The ~20× gap between base-prompts (0.04) and tool-prompts (0.81) is the
quantitative signature of a surgically targeted LoRA: it re-wires
tool-calling behavior without disturbing general knowledge. This matches the
abliterated base's own near-zero first-token KL (0.0001) — the merge adds
capability, not drift.

4. MTP (Multi-Token Prediction)

Well above the ~0.3 threshold where speculative decoding pays off — MTP is
active and beneficial.

Files

Disclaimer

This is a preview built for authorized security research and red-teaming.
It will not refuse and may emit instructions for exploiting systems. Use only
on systems you own or are explicitly authorized to test. Not for
misuse/harmful activity. Apache-2.0, inherited from the base.