pretty_name: Permission Command Corpus
language:

• en
  license: apache-2.0
  task_categories:
• text-classification
  task_ids:
• text-classification
  size_categories:
• 10K<n<100K
  configs:
• config_name: gold
  data_files:
  • split: train
    path: gold/train.csv
  • split: validation
    path: gold/validation.csv
  • split: test
    path: gold/test.csv
• config_name: silver_weak_labels
  data_files:
  • split: raw
    path: silver_weak_labels/raw.csv
• config_name: review_queue
  data_files:
  • split: raw
    path: review_queue/raw.csv

Permission Command Corpus

Three views of a command-safety corpus, for local command-risk classification in
front of an LLM or a tool bridge.

• gold: trusted rows, 321 in total across three splits
• silver_weak_labels: mined weak-label rows from Sigma, LOLBAS, GTFOBins,
  Atomic Red Team and Falco, kept as useful but not promoted to gold
• review_queue: unresolved rows that should not be treated as trusted
  training data

Read this before training on it

Findings from 30 August 2026, measured on the gold config in this repository.

The two classifiers trained on this corpus were degenerate and have been
withdrawn. Each emitted the same score for every input, so the reported ROC AUC
of 0.5000 is what a constant scores rather than a real reading. The cause was
feature scaling, and cowWhySo/permission-gate-onnx carries the measurement, the
cause, and the corrected training scripts. With bounded lexical features and
alpha=1e-4, the same pipeline reaches test ROC AUC 0.9972.

That 0.9972 does not transfer. Held out by generator, it falls to 0.7010.
181 of the 246 training rows come from one model, and much of the headline
number is the classifier recognizing that model's phrasing.

The splits are not consistently stratified. The train split is 50.4% bad and
the test split is 32.5% bad. A test set of 40 rows also carries a confidence
interval wide enough that small differences between models are not measurable
here.

What this corpus does not answer

The corpus grades whether a command is hazardous. An agent permission system
built on a positive allowlist asks a different question, which is whether a
command evades the allowlist.

That difference was measured against a real gate. Given a permission engine that
auto-approves only single simple invocations of known read and build commands, a
hazard model trained on this corpus added zero true positives, because the
allowlist already refused every evasion in the test corpus before the model was
consulted. It produced one to three false positives at every threshold, and
python3 -m pytest tests/ -q scored 1.000, so no threshold removed it.

A corpus aimed at that question needs rows covering what an allowlist actually
lets through. Examples are cargo run, npm install and its postinstall
scripts, make, and git push.

Reasonable uses

• Training and evaluating command-hazard classifiers, with a source-held-out split
• A starting point for a corpus aimed at a specific permission system
• Adjudication and augmentation work, using the silver and review configs

Uses to avoid

• Gating command execution in production on the gold config alone
• Quoting the pooled validation or test numbers without the held-out number beside them
• Merging silver or review rows into gold without adjudication

Gold columns

text, normalized_text, actual_label, surface_label, obfuscation_label,
obfuscation_type, risk_class, shell_family, source, split

Silver and review columns

sample_id, text, normalized_text, label, surface_label,
obfuscation_label, risk_class, label_source, source_family,
source_repo, source_path, source_title, category, attack_tags,
platform, confidence, needs_adjudication, keep_for_training,
training_role, adjudication_reason, adjudication_model, dataset_role,
promotion_status, created_at

Row counts