license: mit
task_categories:

• text-classification
  language:
• en
  tags:
• data-poisoning
• adversarial-machine-learning
• security
• backdoor-attacks
• benchmark
• unrelabel
  pretty_name: unrelabel poison benchmark
  configs:
• config_name: trigger-backdoor
  data_files:
  • split: poisoned_train
    path: trigger-backdoor/poisoned_train.csv
  • split: clean_train
    path: trigger-backdoor/clean_train.csv
  • split: test
    path: trigger-backdoor/test.csv
• config_name: unicode-zero-width
  data_files:
  • split: poisoned_train
    path: unicode-zero-width/poisoned_train.csv
  • split: clean_train
    path: unicode-zero-width/clean_train.csv
  • split: test
    path: unicode-zero-width/test.csv
• config_name: style-register
  data_files:
  • split: poisoned_train
    path: style-register/poisoned_train.csv
  • split: clean_train
    path: style-register/clean_train.csv
  • split: test
    path: style-register/test.csv
• config_name: subpopulation-phone
  data_files:
  • split: poisoned_train
    path: subpopulation-phone/poisoned_train.csv
  • split: clean_train
    path: subpopulation-phone/clean_train.csv
  • split: test
    path: subpopulation-phone/test.csv
• config_name: composite-cooccurrence
  data_files:
  • split: poisoned_train
    path: composite-cooccurrence/poisoned_train.csv
  • split: clean_train
    path: composite-cooccurrence/clean_train.csv
  • split: test
    path: composite-cooccurrence/test.csv
• config_name: availability-noise
  data_files:
  • split: poisoned_train
    path: availability-noise/poisoned_train.csv
  • split: clean_train
    path: availability-noise/clean_train.csv
  • split: test
    path: availability-noise/test.csv

unrelabel poison benchmark

Heads up, it is poison. These files contain deliberately poisoned training data, published for defensive research, benchmarking, and teaching. Point them at your own models to find out what your defenses miss. Do not point them at systems you do not own or have permission to test. Be the red team, not the villain.

Six ways to poison a text classifier, each shipped as a matched pair: the clean training set, the poisoned one, a reversible manifest of exactly what changed, and the behavioral canary that catches it. Built with unrelabel.

The point of the benchmark is the last column. Accuracy is one number, and it lies. Five of these six attacks keep global accuracy perfectly healthy while quietly turning the model's verdict. Only a behavioral canary notices. The sixth (availability) is loud on purpose, so you can see the difference.

Blue is the global test accuracy your dashboard reports; it barely moves, and four attacks leave it identical at 0.942. Red is what actually happened on the exact slice each attack aimed at. Each red marker measures a different named population, so read each row against its own blue dot, not against the other rows. For the four backdoors red is 1 minus ASR (the share of triggered inputs NOT flipped to the attacker label). Availability is the honest control: it is the one attack global accuracy actually catches, so its dots coincide in green.

Stealth is how blind a plain accuracy dashboard is to the attack: 100% means it sees nothing. Pick a config from the dropdown above to browse the poisoned rows, or open a folder for its full card, manifest, and canary.

How to actually use this

pip install unrelabel
# each folder ships a canary.yaml; assert it against your retrained model:
unrelabel check your_config.yaml --canary subpopulation-phone/canary.yaml

Every number here is a measured run, reproducible from the manifests. The tool, the attacks, and the defenses: https://github.com/oz9un/unrelabel.