license: other
task_categories:

• text-retrieval
• text-generation
• question-answering
• summarization
• feature-extraction
  language:
• en
  tags:
• security
• red-team
• redteam
• penetration-testing
• pentesting
• web-security
• cloud-security
• active-directory
• kubernetes
• dfir
• cheatsheets
• wiki
• markdown
• jsonl
• cybersecurity
• cyber-security
• dataset
• retrieval
  pretty_name: HackTricks SlimPajama Corpus
  size_categories:
• n<1K
  dataset_info:
  config_name: hacking_tricks
  features:
  • name: text
    dtype: string
  • name: meta
    struct:
    • name: source
      dtype: string
    • name: source_repo
      dtype: string
    • name: repo_branch
      dtype: string
    • name: repo_commit
      dtype: string
    • name: source_path
      dtype: string
    • name: source_url_github
      dtype: string
    • name: source_url_site
      dtype: string
    • name: title
      dtype: string
    • name: section_titles
      list: string
    • name: license
      dtype: string
    • name: n_chars
      dtype: int64
    • name: n_words
      dtype: int64
    • name: sha256
      dtype: string
    • name: crawl_ts
      dtype: string
    • name: lang
      dtype: string
      splits:
  • name: train
    num_bytes: 9726009
    num_examples: 888
    download_size: 4839940
    dataset_size: 9726009
    configs:
• config_name: hacking_tricks
  data_files:
  • split: train
    path: hacking_tricks/train-*

HackTricks SlimPajama Corpus (Parquet)

A high‑quality, SlimPajama‑style corpus distilled from the HackTricks knowledge base (core wiki and, optionally, HackTricks Cloud). Each record corresponds to a cleaned Markdown article with headings preserved, code fences kept, and rich provenance metadata for reproducible research and downstream retrieval.

Token count: ~2.75M tokens.

⚠️ Ethical use only. This corpus is intended for lawful research, education, and defensive security. Do not use it to violate terms of service or applicable laws.

What’s in this release (Parquet)

• Primary delivery = Parquet shards under data/hacking_tricks/.../train-*.parquet (Hub‑managed) for fast streaming with 🤗 datasets.
• Raw JSONL retained at raw/hacking-tricks.jsonl[.gz|.zst] for transparency and reproducibility.
• Named config: hacking_tricks (single config; no all config).

Highlights

• Clean, article‑level records suitable for pretraining/continued‑pretraining, RAG, and eval.
• Code blocks preserved (fenced code kept as‑is) while removing front‑matter/boilerplate.
• Provenance: repo, commit, path, GitHub blob URL, and a best‑effort canonical site URL per article.
• Lightweight schema: {"text": ..., "meta": {...}} with stable keys for training and retrieval.

Dataset layout

/                           # dataset root (this README)
  raw/
    hacking-tricks.jsonl    # original export (optionally .gz/.zst)
  data/
    hacking_tricks/
      default/1.0.0/train/
        train-00000-of-XXXX.parquet
        train-00001-of-XXXX.parquet
        ...

Depending on future pushes, you may also publish separate configs (hacktricks_core, hacktricks_cloud). This card documents the current single hacking_tricks config.

Schema

Each row is a UTF‑8 JSON object with the following structure (identical for Parquet features):

{
  "text": "<cleaned article markdown with code fences>",
  "meta": {
    "source": "HackTricks (git)",
    "source_repo": "https://github.com/HackTricks-wiki/hacktricks",
    "repo_branch": "master",
    "repo_commit": "<git-sha>",
    "source_path": "path/in/repo/file.md",
    "source_url_github": "https://github.com/.../blob/<sha or branch>/path.md",
    "source_url_site": "https://book.hacktricks.wiki/<page>",
    "title": "<article title>",
    "section_titles": ["H1/H2/H3…"],
    "n_chars": 12345,
    "n_words": 2345,
    "sha256": "<content-hash>",
    "crawl_ts": "2025-11-04T00:00:00+00:00",
    "lang": "en",
    "license": "CC BY-NC-SA 4.0 (per upstream repo)"
  }
}

Field notes

• text keeps fenced code blocks and normalized headings. Front‑matter is removed.
• source_url_site is a best‑effort canonical link to the rendered wiki page; source_url_github always points to the exact file/commit.
• license mirrors the upstream license hint for clarity.

Quickstart (🤗 Datasets)

Load the Parquet config (recommended)

from datasets import load_dataset

REPO = "tandevllc/hacking-tricks"
ds = load_dataset(REPO, name="hacking_tricks", split="train")
print(len(ds), ds[0]["meta"]["title"])  # sample

Direct JSONL loading from raw/ (explicit)

from datasets import load_dataset

ds = load_dataset(
    "json",
    data_files="raw/hacking-tricks.jsonl",
    repo_id="tandevllc/hacking-tricks",
    split="train",
)

Typical filters

# Long English articles only
long_en = ds.filter(lambda r: r["meta"].get("lang") == "en" and r["meta"].get("n_words", 0) > 800)

# Articles with code (simple heuristic)
with_code = ds.filter(lambda r: "```" in (r["text"] or ""))

# Slice by section heading keyword
ad_ops = ds.filter(lambda r: any("Active Directory" in h for h in r["meta"].get("section_titles", [])))

RAG‑ready retrieval columns

• meta.source_url_github — stable pointer to the exact file.
• meta.source_url_site — user‑friendly URL for UI previews.
• meta.sha256 — robust dedup key across updates.

Intended uses

• Pretraining / Continued pretraining for security‑aware models.
• RAG / QA over a curated set of red‑team/blue‑team encyclopedic pages.
• Summarization & explanation of workflows, checklists, and attack surfaces.
• Evaluation: security knowledge, long‑form reasoning on procedures, and code‑aware contexts.

This corpus is didactic/encyclopedic. It does not replace hands‑on lab data, exploit PoCs, or live telemetry.

Cleaning & quality

• YAML front‑matter and boilerplate removed; headings normalized; excessive blank lines collapsed.
• Code fences retained verbatim for downstream code‑aware models.
• Article‑level minimum length applied (very short files excluded).
• Stable dedup via meta.sha256 and exact source_path at a specific commit.

Limitations & caveats

• Upstream drift: HackTricks evolves; future updates may add/rename pages.
• License constraints: Non‑commercial, ShareAlike terms apply; see License below.
• Context gaps: Some pages assume prior knowledge or external links.
• No execution traces: Articles describe tactics; they are not runtime logs or CVE feeds.

License

• Underlying content: © respective HackTricks authors, typically CC BY‑NC‑SA 4.0 as stated in the upstream repositories/site.
• License: "TanDev Proprietary License — All Rights Reserved"

When you use this dataset, you must:

1. Provide attribution to HackTricks and the specific article(s) used.
2. Use it non‑commercially unless you have additional permission from rights holders.
3. Distribute derivatives under the same license (ShareAlike).

If you are a rights holder and would like a page revised or removed, please open an issue on the dataset page or email the maintainer.

Citation

If this dataset helps your research, please cite it alongside HackTricks:

@dataset{tandevllc_2025_hacktricks_slimpajama,
  author = {Gupta, Smridh},
  title  = {HackTricks SlimPajama Corpus},
  year   = {2025},
  url    = {https://huggingface.co/datasets/tandevllc/hacking-tricks}
}

You should also attribute individual HackTricks pages used (e.g., in an acknowledgments section) to satisfy CC BY‑NC‑SA 4.0.

Maintainer

Smridh Gupta — [email protected]