Get the app
extract gated 0 0

tandevllc/offsec_redteam_info

published by @tandevllc

Alignment dataset tracked in the /datasets sub-catalog.

Lifetime downloads
444
Last 30 days
30
Likes
3
Size
10.0 GB
Created on HF
2025-11-02
Age
11 months ago

Description

OffSec RedTeam Info

OffSec RedTeam Info is a SlimPajama‑style, category‑organized corpus of security knowledge text crawled from reputable red‑team/blue‑team websites: wikis, training blogs, vendor research, CERT advisories, reversing/malware labs, cloud/kubernetes posts, OSINT handbooks, AD tradecraft, and more.

Token count: ~1.646B tokens.

⚠️ Ethical use only. Use for research, education, and defensive security. Respect robots.txt, site terms, and copyrights. Do not misuse… See the full description on the dataset page: https://huggingface.co/datasets/tandevllc/offsec_redteam_info.

Tags

task_categories:text-generationtask_categories:text-retrievaltask_categories:text-rankingtask_categories:feature-extractiontask_categories:sentence-similaritytask_categories:question-answeringtask_categories:summarizationlanguage:enlicense:othersize_categories:1M<n<10Mformat:parquetmodality:textlibrary:datasetslibrary:dasklibrary:mlcroissantlibrary:polarsregion:ussecurityred-teamredteamoffensive-securityoffsecpentestingpenetration-testingosintdfirthreat-intelcloud-securitykubernetesactive-directorymalware-analysisreversingtraining-blogswebsecurityweb-securitydatasetjsonlparquetcybersecuritycyber-security

README current version from Hugging Face

pretty_name: OffSec RedTeam Info
license: other
task_categories:

  • text-generation
  • text-retrieval
  • text-ranking
  • feature-extraction
  • sentence-similarity
  • question-answering
  • summarization
    language:
  • en
    tags:
  • security
  • red-team
  • redteam
  • offensive-security
  • offsec
  • pentesting
  • penetration-testing
  • osint
  • dfir
  • threat-intel
  • cloud-security
  • kubernetes
  • active-directory
  • malware-analysis
  • reversing
  • training-blogs
  • websecurity
  • web-security
  • dataset
  • jsonl
  • parquet
  • cybersecurity
  • cyber-security
    size_categories:
  • 1M<n<10M
    configs:
  • config_name: ad_ops
    data_files:
    • split: train
      path: ad_ops/train-*
  • config_name: binary_exploitation
    data_files:
    • split: train
      path: binary_exploitation/train-*
  • config_name: c2_tradecraft
    data_files:
    • split: train
      path: c2_tradecraft/train-*
  • config_name: cloud_redteam
    data_files:
    • split: train
      path: cloud_redteam/train-*
  • config_name: core_wikis
    data_files:
    • split: train
      path: core_wikis/train-*
  • config_name: dfir_detection
    data_files:
    • split: train
      path: dfir_detection/train-*
  • config_name: ics_scada
    data_files:
    • split: train
      path: ics_scada/train-*
  • config_name: kubernetes_container
    data_files:
    • split: train
      path: kubernetes_container/train-*
  • config_name: linux_unix
    data_files:
    • split: train
      path: linux_unix/train-*
  • config_name: mobile_wireless
    data_files:
    • split: train
      path: mobile_wireless/train-*
  • config_name: osint_recon
    data_files:
    • split: train
      path: osint_recon/train-*
  • config_name: password_cracking
    data_files:
    • split: train
      path: password_cracking/train-*
  • config_name: phishing_se
    data_files:
    • split: train
      path: phishing_se/train-*
  • config_name: reversing_malware
    data_files:
    • split: train
      path: reversing_malware/train-*
  • config_name: threat_intel
    data_files:
    • split: train
      path: threat_intel/train-*
  • config_name: training_blogs
    data_files:
    • split: train
      path: training_blogs/train-*
  • config_name: web_app
    data_files:
    • split: train
      path: web_app/train-*
  • config_name: windows_privesc
    data_files:
    • split: train
      path: windows_privesc/train-*
      dataset_info:
  • config_name: ad_ops
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 90620904
      num_examples: 13277
      download_size: 37244413
      dataset_size: 90620904
  • config_name: binary_exploitation
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 47008528
      num_examples: 3079
      download_size: 22074655
      dataset_size: 47008528
  • config_name: c2_tradecraft
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 2834407
      num_examples: 554
      download_size: 1327232
      dataset_size: 2834407
  • config_name: cloud_redteam
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 2547733802
      num_examples: 270351
      download_size: 900519483
      dataset_size: 2547733802
  • config_name: core_wikis
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 368030997
      num_examples: 44160
      download_size: 82175469
      dataset_size: 368030997
  • config_name: dfir_detection
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 541381934
      num_examples: 90481
      download_size: 191197045
      dataset_size: 541381934
  • config_name: ics_scada
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 243297943
      num_examples: 26862
      download_size: 88171425
      dataset_size: 243297943
  • config_name: kubernetes_container
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 555091963
      num_examples: 63232
      download_size: 210104862
      dataset_size: 555091963
  • config_name: linux_unix
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 1045679383
      num_examples: 89682
      download_size: 388247604
      dataset_size: 1045679383
  • config_name: mobile_wireless
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 1007349882
      num_examples: 95645
      download_size: 315184952
      dataset_size: 1007349882
  • config_name: osint_recon
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 2351643
      num_examples: 850
      download_size: 1085598
      dataset_size: 2351643
  • config_name: password_cracking
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 340784203
      num_examples: 141474
      download_size: 116324635
      dataset_size: 340784203
  • config_name: phishing_se
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 374751470
      num_examples: 25199
      download_size: 127489943
      dataset_size: 374751470
  • config_name: reversing_malware
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 1086556625
      num_examples: 156629
      download_size: 444408093
      dataset_size: 1086556625
  • config_name: threat_intel
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 1283368209
      num_examples: 204353
      download_size: 441308074
      dataset_size: 1283368209
  • config_name: training_blogs
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 222311513
      num_examples: 18974
      download_size: 103640197
      dataset_size: 222311513
  • config_name: web_app
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 1350432930
      num_examples: 202902
      download_size: 481909682
      dataset_size: 1350432930
  • config_name: windows_privesc
    features:
    • name: text
      dtype: string
    • name: meta
      struct:
      • name: url
        dtype: string
      • name: title
        dtype: string
      • name: source
        dtype: string
      • name: category
        dtype: string
      • name: timestamp
        dtype: string
      • name: language
        dtype: string
        splits:
    • name: train
      num_bytes: 27421179
      num_examples: 5747
      download_size: 13600773
      dataset_size: 27421179

OffSec RedTeam Info

OffSec RedTeam Info is a SlimPajama‑style, category‑organized corpus of security knowledge text crawled from reputable red‑team/blue‑team websites: wikis, training blogs, vendor research, CERT advisories, reversing/malware labs, cloud/kubernetes posts, OSINT handbooks, AD tradecraft, and more.

Token count: ~1.646B tokens.

⚠️ Ethical use only. Use for research, education, and defensive security. Respect robots.txt, site terms, and copyrights. Do not misuse this corpus to harm systems or violate laws.

What’s new (Nov 2025)

  • Per‑category Parquet configs published for fast streaming via datasets.load_dataset(...).
  • raw/** JSONL** kept alongside Parquet for reproducibility and low‑level processing.
  • Consistent schema across all categories (see below) with required text and meta keys.
  • Balanced shard sizes (≈256–512 MB) to keep memory steady during load & push.

Repository layout

/                          # dataset root (this card lives here as README.md)
  raw/                     # line-delimited JSON for each category (post-clean)
    <category>.jsonl       # e.g., raw/ad_ops.jsonl
  <category>/              # per-category Parquet config (train split)
    <category>.parquet     # e.g., ad_ops/ad_ops.parquet

Parquet is present for non‑empty categories. Some categories may be JSONL‑only depending on the snapshot.

Categories

  • core_wikis – foundational red‑team/blue‑team references (ATT&CK, CAPEC/CWE, GTFOBins, LOLBAS, PayloadsAllTheThings, etc.).
  • web_app – OWASP content, web vulns, API security, web‑sec blogs.
  • training_blogs – walkthroughs, labs, CTF‑style posts and methodology.
  • ad_ops – Active Directory/Windows internals, abuse paths, domain tradecraft.
  • windows_privesc, linux_unix – OS‑specific privilege escalation & hardening.
  • cloud_redteam, kubernetes_container – cloud & container security.
  • osint_recon, phishing_se – OSINT techniques, social engineering.
  • c2_tradecraft – C2 techniques, operator tradecraft (defensive write‑ups included).
  • mobile_wireless – mobile, Wi‑Fi/Bluetooth/802.11 and radio‑adjacent topics.
  • ics_scada – industrial control systems / OT security.
  • reversing_malware – reversing & malware analysis posts from labs and vendors.
  • binary_exploitation – pwn, exploitation notes, vuln research.
  • password_cracking – hashcat/john guides, NIST/NCSC guidance.
  • dfir_detection – incident response, detection engineering, Sigma, DFIR reports.
  • threat_intel – vendor TI, advisories, newsroom items with technical depth.

Exact category availability depends on the current revision (feeds change; some snapshots may be sparser).

Schema (UPDATED)

Each record follows exactly this structure:

{
  "text": "<cleaned article/content text>",
  "meta": {
    "url": "https://example.com/path",
    "title": "<page title>",
    "source": "example.com",
    "category": "ad_ops",
    "timestamp": "2025-11-02T22:21:39.384421+00:00",
    "language": "en"
  }
}

Field definitions

  • text (string) — cleaned article/content text (readability‑style extraction, normalized whitespace).

  • meta (object) — metadata container with the following keys:

    • url (string) — canonical URL of the item.
    • title (string|null) — page title.
    • source (string|null) — site/domain the content came from (e.g., www.semperis.com).
    • category (string) — logical bucket matching the config name (e.g., ad_ops).
    • timestamp (string, ISO‑8601) — fetch/process time for the item.
    • language (string) — language code (e.g., en).

The sample you shared from Semperis conforms to this schema.

Load examples

Load a single category (Parquet, recommended)

from datasets import load_dataset

REPO = "tandevllc/offsec_redteam_info"
cat = "web_app"  # pick any listed config

ds = load_dataset(REPO, name=cat, split="train")
print(len(ds), ds.column_names[:6])
print(ds[0]["text"][:400])
print(ds[0]["meta"]["url"])  # access metadata

Load multiple categories and interleave

from datasets import load_dataset, interleave_datasets

REPO = "tandevllc/offsec_redteam_info"
names = ["core_wikis", "training_blogs", "threat_intel"]
parts = [load_dataset(REPO, name=n, split="train") for n in names]

# Uniform interleave (good for blended training/eval)
blend = interleave_datasets(parts, probabilities=[1/len(parts)]*len(parts), seed=42)

Filter typical research slices

# Keep only long English articles
long_en = ds.filter(lambda r: (r.get("text") and len(r["text"]) > 1200) and ((r.get("meta") or {}).get("language") == "en"))

# Narrow to a specific source/domain
from_portswigger = ds.filter(lambda r: ((r.get("meta") or {}).get("source") or "").endswith("portswigger.net"))

Load raw JSONL

from datasets import load_dataset

raw = load_dataset(
    "json",
    data_files="raw/web_app.jsonl",
    repo_id="tandevllc/offsec_redteam_info",
    split="train",
)

Cleaning & quality (high level)

  • Content extracted with readability‑style heuristics; multi‑block merge when the best block is short.
  • Basic quality gates: minimum words/sentences, alpha‑fraction, optional index‑page filtering by link density.
  • Normalization: canonicalized URLs, per‑category dedup by link/content hash (some snapshots may apply global dedup).
  • Non‑content and noisy paths avoided (search, feeds, asset dirs, etc.).

These heuristics favor clean prose and technical material, but may still include boilerplate or miss embedded code blocks.

Intended uses

  • Pretraining / continued pretraining of security‑aware language models.
  • RAG / retrieval over current security references and blogs, by category/site.
  • Evaluation of security knowledge, extraction, summarization, and long‑context QA.
  • Trend analysis across sources (pair with timestamps when present).

This dataset is not a CVE ground‑truth database and does not replace vendor advisories.

Limitations & caveats

  • Copyrights & terms apply. Underlying website content retains the publisher’s license/terms.
  • Temporal drift. Websites change; snapshots may vary; links can rot.
  • Extraction noise. Readability may omit figures/code or include navigation text.
  • Metadata sparsity. Some fields are missing for certain sources.

License & access

License: "TanDev Proprietary License — All Rights Reserved"

Underlying content: remains under each site’s terms. For conservative use, store only links and your own embeddings/summaries.

Commercial usage: A paid TanDev Commercial License is available for commercial training/inference and internal derivatives. Contact [email protected] with organization, intended use, and deployment details.

Takedowns: If you own content included here and want it removed, please open an issue or email the maintainer.

Citation

@dataset{tandevllc_2025_offsec_redteam_info,
  author = {Gupta, Smridh},
  title  = {OffSec RedTeam Info},
  year   = {2025},
  url    = {https://huggingface.co/datasets/tandevllc/offsec_redteam_info}
}

Maintainer

Smridh Gupta — [email protected]

README history 20 revisions

Every night we snapshot the README of every dataset in the catalog. When the SHA changes we archive the new version and diff it against the last. This is the evolving thought record of the alignment-data field: what the author decided to say about the corpus, and how that framing shifted over time.

  1. 2025-11-06Update README.md4e0536317.7 KB
    Loading...
  2. 2025-11-06Update README.md016550617.7 KB
    Loading...
  3. 2025-11-06Update README.mdce298a117.7 KB
    Loading...
  4. 2025-11-06Add 'windows_privesc' config from local windows_privesc.jsonl (5747 rows)dd20b8217.7 KB
    Loading...
  5. 2025-11-06Add 'osint_recon' config from local osint_recon.jsonl (850 rows)793eded17.2 KB
    Loading...
  6. 2025-11-06Add 'c2_tradecraft' config from local c2_tradecraft.jsonl (554 rows)0d7365216.7 KB
    Loading...
  7. 2025-11-06Add 'binary_exploitation' config from local binary_exploitation.jsonl (3079 r...8e8e8dd16.1 KB
    Loading...
  8. 2025-11-06Update README.mdc6e371c15.6 KB
    Loading...
  9. 2025-11-06Delete windows_privesc34c6ed817.7 KB
    Loading...
  10. 2025-11-06Delete osint_recon37fd84617.7 KB
    Loading...
  11. 2025-11-06Delete c2_tradecraftcdac21517.7 KB
    Loading...
  12. 2025-11-06Delete binary_exploitation6a1b49c17.7 KB
    Loading...
  13. 2025-11-06Upload 4 filesc3c882417.7 KB
    Loading...
  14. 2025-11-06Delete raw/binary_exploitation.jsonl1fc1f9517.7 KB
    Loading...
  15. 2025-11-06Delete raw/c2_tradecraft.jsonl81fd0dc17.7 KB
    Loading...
  16. 2025-11-06Delete raw/osint_recon.jsonla1e292917.7 KB
    Loading...
  17. 2025-11-06Delete raw/windows_privesc.jsonl9658cf017.7 KB
    Loading...
  18. 2025-11-06Add 'reversing_malware' config from local reversing_malware.jsonl (156629 rows)0a10d1117.7 KB
    Loading...
  19. 2025-11-06Update README.mdaf4393117.2 KB
    Loading...
  20. 2025-11-06Upload reversing_malware.jsonl20d9f0f17.7 KB
    Loading...

Used by abliterated models none yet

No abliterated models in our catalog have declared this dataset in their YAML metadata yet. This can mean the dataset is used but not declared, is used outside abliteration workflows, or is new. The link will populate automatically as models are indexed.

Community discussions 1 thread · 1 open · 1 comments

Threads posted by the community on Hugging Face: bug reports, questions about usage, pull requests on the README, license clarifications. Click any thread to expand and read the full conversation inline. Newest first.

  1. 2025-11-06[bot] Conversion to Parquetopen1 💬#1
    Loading...

Continue on Hugging Face →

Related datasets same stage · extract

Anthropic/hh-rlhf
2.1M dl 2442 likes 90.4 MB
Dataset Card for HH-RLHF Dataset Summary This repository provides access to two different kinds of data: Human ...
djain95/sae-jailbreaks-cache
468.6K dl 1 likes 30.6 GB
mosaicml/dolly_hhrlhf
309.5K dl 113 likes 23.7 MB
Dataset Card for "dolly_hhrlhf" This dataset is a combination of Databrick's dolly-15k dataset and a filtered subset of Anthrop...
Dahoas/full-hh-rlhf
266.3K dl 104 likes 129.8 MB
Dataset Card for "full-hh-rlhf" Anthropic's HH dataset reformatted into prompt, chosen, rejected samples.
mlabonne/harmful_behaviors
198.2K dl 169 likes 23 KB
mlabonne/harmless_alpaca
190.2K dl 51 likes 1.2 MB
walledai/AdvBench
176.3K dl 123 likes 38 KB
Dataset Card for AdvBench Paper: Universal and Transferable Adversarial Attacks on Aligned Language Models Data: AdvBench Datas...
JailbreakV-28K/JailBreakV-28k
116.4K dl 80 likes 288.8 MB
⛓‍💥 JailBreakV-28K: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks 🌐 ...

Read further

Community best practices (extraction)
5 rules the field converged on for building the pair.
Building your own extraction pair
When the reference pair is not enough. Sizing, framing, and evaluation.
Datasets that shaped abliteration
The five corpora that did most of the work.
Non-English abliteration communities
How Chinese, Russian, Japanese, and Korean practitioners build extraction pairs.

Metadata

License
other

Repository

DIR ad_ops DIR binary_exploitation DIR c2_tradecraft DIR cloud_redteam 10 files DIR core_wikis 2 files DIR dfir_detection 3 files DIR ics_scada DIR kubernetes_container 3 files DIR linux_unix 5 files DIR mobile_wireless 4 files DIR osint_recon DIR password_cracking 2 files DIR phishing_se 2 files DIR raw 14 files DIR reversing_malware 5 files DIR threat_intel 6 files DIR training_blogs DIR web_app 6 files DIR windows_privesc FILE .gitattributes FILE README.md

File contents live on Hugging Face. Click a file or folder to open it there.

Download the dataset

Gated dataset. You need to accept the author's terms on Hugging Face before downloading. Visit the repository, click Agree and access repository, then log in locally with huggingface-cli login and use one of the snippets below.

Copy any of these snippets into your notebook or terminal. All three fetch directly from Hugging Face using your own credentials.

Recommended: datasets library (Python) 
from datasets import load_dataset
ds = load_dataset("tandevllc/offsec_redteam_info")
Raw snapshot (Python) 
from huggingface_hub import snapshot_download
snapshot_download(repo_id="tandevllc/offsec_redteam_info", repo_type="dataset")
git clone (requires git-lfs) 
git clone https://huggingface.co/datasets/tandevllc/offsec_redteam_info
Browse repository on Hugging Face ->
Catalog is the map. Apps are the tools.

Run models on your own machine, not in the cloud.

Every model page has an "Open in Abliteration" button that hands the model directly to the first-party desktop client, at the quantization your rig can actually run. No API keys, no subscription, no prompt leakage.

Get the app Browse the catalog
Open in Abliteration