license: other
license_name: mixed-see-card
language:

• en
  task_categories:
• text-generation
  size_categories:
• 10B<n<100B
  dataset_curators:
• Arkadiusz Słota (SlayerLab)
  pretty_name: Slayer139 pretraining mix (tok24k)
  tags:
• pretraining
• tokenized

Slayer139 pretraining mix (tok24k)

Summary

A pre-tokenized English pretraining mix: 26 882 183 219 tokens in 24 binary shards of uint16 token ids,
produced with the tok24k tokenizer (vocab 24 576, included as tokenizer.json, sha256 prefix afeb22b4).
It was used to train the Slayer139 model (139 279 760 parameters). No quality or benchmark claims are made here.

The release contains only token ids, not raw text. Token ids can be decoded back to text, so the licences of
the upstream sources still apply (see Licensing).

Files: shards/s00.bin–shards/s07.bin, shards/t00.bin–shards/t15.bin, tokenizer.json, README.md, SHA256SUMS.
SHA-256 of the concatenation of all shards in order s00…s07, t00…t15: prefix 246e7b83 (full hash in SHA256SUMS).

Format

• Each shard is a flat little-endian uint16 array of token ids (no header).
• Documents are concatenated and separated by the EOS token <|endoftext|>, id 0.
• Global token offsets (lo–hi, half-open) are given in the shard table below.

import numpy as np
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
ids = np.memmap("shards/t00.bin", dtype=np.uint16, mode="r")
eos = np.flatnonzero(ids[:1_000_000] == 0)
doc = ids[eos[0] + 1 : eos[1]].tolist()  # first complete document in the shard
print(tok.decode(doc))

Composition

Two parts:

1. arcmix24k (shards s00–s07, 8 615 887 241 tokens): a base pool re-tokenized with tok24k. The pool is
   SlayerLab/minimal-en-corpus-5b@bb3aec23 plus new FineWeb-Edu (100BT sample) documents, OpenStax textbooks
   (×4 copies), and ARC-related upweight copies, minus 5 537 documents (4 362 removed by the decontamination scan,
   1 175 removed because they were CC BY-NC-SA).
2. mix24k (shards t00–t15, 18 266 295 978 tokens): a top-up from 14 Hugging Face / open-data sources, exact-deduplicated
   against the base pool and against FineWeb-Edu files already used, interleaved with arcmix24k shards s06–s07
   (2 153 971 445 tokens) — i.e. those two shards' documents occur in both parts.

Per-source tokens in the top-up part (tok24k, billions). "Declared licence" is the license field of the source's
Hugging Face metadata at the pinned revision (as read on 2026-10-12); "—" means no licence is declared there.

The OpenSubtitles source was planned but never fetched; it is not included.

Shards

The Slayer139 training rotation used 22 shards (s00–s05 + t00–t15 = 23 670 620 160 tokens) at 32 835 steps per shard;
s06–s07 are published for completeness.

Intentional duplication

This mix is not globally deduplicated. Known intentional duplicates:

• ARC-related documents (web documents related to ARC-style science questions) are duplicated 2–3×. Within
  arcmix24k about 1.13 B tokens are intentional duplicate copies; the FineWeb-Edu expansion stream contains
  1 480 815 intentional ARC-related copies (≈1.89 B tokens).
• OpenStax textbooks appear 4 times.
• arcmix24k shards s06–s07 are also mixed into the top-up part.

Decontamination

Scope is narrow; please read precisely.

• Checked against WikiText-2 and ARC-Easy / ARC-Challenge only, using 13-gram overlap plus exact matching of short questions.
• After cleaning: 0 hits against the WikiText-2 test and validation splits and the ARC-Easy/ARC-Challenge test and
  validation splits; positive controls confirmed the scanner detects planted overlap.
• The top-up stage additionally removed 25 808 documents (union of: 13-gram hits against WikiText-2 and ARC-Easy/Challenge
  test+validation; documents sharing ≥ 10 distinct 13-grams with a control document; documents containing a BLiMP sentence
  of ≥ 6 words).
• The decontamination statement in the base corpus card (SlayerLab/minimal-en-corpus-5b) does not apply to this release;
  the scans above supersede it.
• 467 documents overlapping ARC train questions were kept on purpose.
• Not checked: HellaSwag, PIQA, MMLU, LAMBADA, GSM8K, HumanEval and any other benchmark. BLiMP was scanned only in the
  top-up part (shards t00–t15), not in arcmix24k. Do not assume this data is clean for those.

Licensing

The dataset is released under license: other because it aggregates sources with different terms. The licence of
each upstream source (table above) continues to apply to its part of the data. Users are responsible for complying
with the terms of each source, including gated sources (StarCoderData, tiny-textbooks) whose access conditions are
set by their maintainers. OpenStax material is CC BY 4.0; ARC is CC BY-SA 4.0.

Dataset curators

Arkadiusz Słota (SlayerLab)

Personal information

No PII scrubbing beyond upstream (e.g. FineWeb e-mail/IP anonymisation). A length-weighted random sample of 100,000
documents (575M tokens) contained e-mail-like strings in 2.6% of documents (mostly corresponding-author and
organisational addresses in scientific and news text), phone-like strings in 2.7% (many false positives; real ones
mostly institutional), IPv4-like strings in 1.1% (mostly non-personal). Do not use this data to identify or contact
individuals.

Reproducibility

• All Hugging Face sources are pinned to the commit revisions in the Composition table.
• PMC was read from the AWS Open Data bucket s3://pmc-oa-opendata (inventory snapshot 2026-10-05T01-00Z); this
  part is not bit-reproducible because the bucket changes over time.
• Shard integrity: verify with sha256sum -c SHA256SUMS.

Citation

@misc{slayerlab_pretrain_mix_24k_2026,
  title  = {Slayer139 pretraining mix (tok24k)},
  author = {Arkadiusz Słota (SlayerLab)},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/datasets/SlayerLab/slayer139-pretrain-mix-24k}}
}

Please also cite the upstream datasets.