language:

• en
  license: cc-by-nc-4.0
  size_categories:
• 1K<n<10K
  task_categories:
• text-generation
• text-classification
  tags:
• red-teaming
• safety
• finance
• benchmark
• adversarial
• llm-safety
• financial-risk
  pretty_name: "FinRED: Financial Red-Teaming Benchmark"
  dataset_info:
  features:
  • name: category_prefix
    dtype: string
  • name: prompt
    dtype: string
    splits:
  • name: train
    num_examples: 5805
    configs:
• config_name: default
  data_files:
  • split: train
    path: "data/train-*"

FinRED: An Expert-Guided Red-Teaming Benchmark for Financial LLM Safety

Dataset Description

FinRED is the first comprehensive red-teaming benchmark specifically designed to evaluate the safety of Large Language Models (LLMs) in the financial domain. Developed in collaboration with 12 domain experts from South Korea's Financial Security Institute (FSI), FinRED provides 5,805 adversarial seed prompts grounded in realistic financial threat scenarios.

Unlike general-purpose safety benchmarks that focus on universal harms, FinRED targets domain-specific financial risks — including regulatory compliance violations, financial fraud facilitation, and systemic trust erosion — that require specialized evaluation.

• Paper: FinRED: An Expert-Guided Red-Teaming Benchmark for Financial LLM Safety (KDD 2026)
• Repository: https://github.com/anon-user-7777/FINRED-PAPER

Dataset Summary

Two-Level Financial Risk Taxonomy

FinRED is built upon a novel 2-level financial risk taxonomy established through extensive literature review, focus group interviews (FGI), and consensus-building with financial security experts.

Level-1 Categories

Level-2 Subcategories

Dataset Statistics

Seeds per Category

Seeds per Subcategory

Dataset Structure

Data Fields

• category_prefix (string): The Level-2 taxonomy code (e.g., R1_1, R3_2, R5_7) identifying the financial risk category of the prompt.
• prompt (string): A contextually rich adversarial seed prompt (behavior seed) describing a realistic financial threat scenario.

Data Splits

Dataset Creation

Generation Pipeline

FinRED prompts are generated through a schema-driven, context-grounded pipeline:

1. Financial Risk Taxonomy & Schema Definition: Expert-defined JSON schemas for each risk category, refined through iterative collaboration with FSI experts over multiple review cycles.
2. Contextual Retrieval: ~500 expert-curated financial documents (regulatory frameworks, supervisory guidelines, audit reports) are chunked and indexed in a vector database. Relevant chunks are retrieved per Level-2 category.
3. Context-Grounded Behavior Seed Generation: A two-step process — first generating structured JSON scenarios from schemas and retrieved context, then transforming them into natural-language seed prompts with dynamic persona assignment and strategic element combination.
4. Expert Validation: Generated prompts undergo final validation by financial experts through focus group interviews, evaluating scenario realism, attack effectiveness, and clarity of potential harm.

Regulatory Alignment

The taxonomy is aligned with international supervisory frameworks including:

• FATF (Financial Action Task Force)
• BIS/BCBS (Basel Committee on Banking Supervision)
• NIST Cybersecurity Framework
• OWASP Security Standards
• EU DORA (Digital Operational Resilience Act)
• ISO/IEC 27001 Information Security Management