base_model: orcarouter/Qwen3.8-27B-Uncensored-FP8
library_name: transformers
pipeline_tag: image-text-to-text
license: other
language:
- en
- fr
tags: - unsloth
- lora
- qlora
- cyber
- image-text-to-text
- vision
- video
model-index:
- name: Qwen3.8-27B-Uncensored-FP8-cyber
results:- task:
type: text-generation
name: Causal language modeling
dataset:
name: cyber SFT holdout
type: dataset_cyber.jsonl
metrics:- type: loss
value: 3.998755
name: eval_loss
- type: loss
- task:
Qwen3.8-27B-Uncensored-FP8-cyber
Instruction-tuned cybersecurity assistant (offensive, defensive, GRC, architecture, SOC/DFIR, RSSI).
| Base model | orcarouter/Qwen3.8-27B-Uncensored-FP8 |
| Domain | cyber |
| Method | LoRA / QLoRA (Unsloth) · rank 8 · α 16 |
| Quantization at train | bf16 LoRA |
| Context | 2048 tokens |
| Dataset | dataset_cyber.jsonl · train 84471 / eval 854 |
| GPU | NVIDIA RTX PRO 6000 Blackwell Server Edition (95.0 GiB) |
| Wall time | 456.3 min |
| Modalities kept | vision, video |
This checkpoint continues a strong general model and specialises it on a curated SFT corpus of cybersecurity procedures: pentest / red team, SOC and DFIR, cloud and identity, GRC (ISO, NIST, NIS2, DORA), and RSSI / project-management questions. Answers are meant to be concrete (controls, detections, hardening), not generic essays.
What changed vs the reference
Reference = the published base checkpoint orcarouter/Qwen3.8-27B-Uncensored-FP8, plus the first in-run loss (LoRA ≈ 0 at step 0).
| Metric | Reference (base / first log) | This fine-tune | Δ |
|---|---|---|---|
| Train loss (first → last logged) | 14.2453 | 5.0644 | -64.4% |
| Train loss (best) | — | 3.7237 | — |
| Eval loss (holdout, first → last) | 8.9583 | 3.9988 | -55.4% |
The first logged train loss is the closest in-run proxy for the base model (LoRA starts near zero). Option F, when executed, adds an independent holdout comparison against the frozen merged base.
Training data
- File:
dataset_cyber.jsonl - Split:
0.01holdout, seed 42 - Format: chat-templated SFT (
messages/instruction+output/### Instruction+### Response)
Training procedure
| Hyperparameter | Value |
|---|---|
| Epochs | 1 |
| Learning rate | 0.0002 |
| Warmup ratio | 0.05 |
| Device batch | 4 |
| Grad accum | 2 |
| Effective batch | 8 |
| Optim | adamw_8bit |
| Packing | True |
| LoRA targets | ['q_proj', 'k_proj', 'v_proj', 'o_proj', 'out_proj', 'gate_proj', 'up_proj', 'down_proj'] |
Intended use
Authorized defensive work, tabletop exercises, control design, detection engineering, audit readiness, and explaining attack techniques without weaponized payloads.
Out of scope: Do not use it to attack systems you do not own, to generate exploit payloads, or as a substitute for a licensed auditor or incident commander.
Multimodal
Kept towers: vision, video. Vision/audio layers were frozen during text SFT (vision=False, audio=False). Load the merged Transformers folder (or GGUF + mmproj) to keep image / video / audio.
How to use
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
repo = "nico248000000000/Qwen3.8-27B-Uncensored-FP8-cyber"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
repo, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True,
)
messages = [{"role": "user", "content": 'How do you design a reliable Zero Trust infrastructure? Answer with a concrete control list.'}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
print(tok.decode(model.generate(inputs, max_new_tokens=256)[0], skip_special_tokens=True))
Limitations
- Domain shift: quality drops outside the SFT topics.
- Eval above is holdout loss (and optional targeted checks). It is not a public leaderboard.
- The base model license and acceptable-use policy still apply.
License
other — inherit and respect the license of orcarouter/Qwen3.8-27B-Uncensored-FP8.