mirror

Seems like it could be useful for crafting toxic preference datasets, so I am mirroring it.

AI Village - Generative Red Team 2 Dataset

Data Dictionary

At Defcon 32, the AI Village hosted the 2nd Generative Red Teaming event (GRT2). This dataset contains all the raw data collected during the event.

The dataset is provided in both csv and parquet formats for convenience.

Inference (grt2_inference.csv/grt2_inference.parquet)

Contains all inference requests sent to the OLMo API endpoint during the GRT2 event. All samples contained in runs and/or submissions are included in this dataset, along with any requests that were not part of a submission.

• id: Unique identifier for each inference request.
• timestamp: Timestamp of the request.
• status_code: API status code response to the request (200/400/404).
• request_message: Message sent to the OLMo model.
• response_message: Response from the OLMo model.

Runs (grt2_runs.csv/grt2_runs.parquet)

Contains metadata about runs created by users. Each row represents a unique run that may or may not have been included in a submission.

• run_id: Unique identifier for the run.
• user_id: Identifier of the user who created the run.
• commentary: Additional commentary provided by the user.
• eval_name: Evaluation type chosen by the user when creating the run. Options: 'privacy', 'harmful-language', 'misinformation', 'malicious-uses', or 'custom'.
• submission_ids: List of submission identifiers in which the run was included.
• created_at: Datetime of run creation.
• primary_scorer: Name of the primary scorer.
• primary_score: Value of the primary score.
• secondary_scorer: Type of the secondary scorer (if available).
• secondary_score: Value of the secondary score (if available).

Submissions (grt2_submissions.csv/grt2_submissions.parquet)

Contains all submissions created by users from one or more runs, including information on monetary awards.

• submission_id: Unique identifier for each submission.
• user_id: Identifier of the user who created the submission.
• eval_name: Evaluation type, similar to eval_name in the runs dataset.
• commentary: Additional commentary provided by the user.
• run_ids: One or more run identifiers used in the submission.
• evidence: Any evidence files included by the user.
• created_at: Datetime of submission creation.
• created_type: Either 'API' or 'UI', indicating whether the submission was created using the Crucible user interface (UI) or uploaded via the Crucible web API endpoint.
• status: Either 'draft' if not submitted to Bugcrowd, or 'new' if submitted.
• bugcrowd_id: Unique identifier for the submission in Bugcrowd.
• monetary_amount: Total USD payout for the submission.

Samples (grt2_samples.csv, grt2_samples.parquet)

Contains all request/response pairs linked to a given run with scoring output.

• run_id: Unique identifier of the run that the sample is linked to.
• sample_id: Sample identifier within the run (may not be unique if epoch > 1).
• epoch: If > 1, indicates the same input was run multiple times.
• input: Raw input text.
• response: Response text from the model.
• Score columns (contain a value if the given scoring metric was enabled, otherwise null):
  • crucible/presidio.value: Presidio score value (if available).
  • crucible/wildguard.value: WildGuard score value (if available).
  • crucible/perspective.value: Perspective score value (if available).
  • crucible/zero_shot.value: Score from a zero-shot classifier (if available).
  • model_graded_qa.value: Score from a model-graded QA.
  • crucible/wildguard.metadata.harmful_request: Boolean value indicating if identified as a harmful request by WildGuard.
  • crucible/wildguard.metadata.refusal_response: Boolean value indicating if identified as a refusal response by WildGuard.
  • crucible/wildguard.metadata.harmful_response: Boolean value indicating if response identified as harmful by WildGuard.