license: cc-by-4.0
pretty_name: BFSI-Bench
extra_gated_fields:
Name: text
Email: text
Organization / Institution: text
Role: text
Country: country
Intended use:
type: select
options:
- Academic research
- Commercial research
- Model evaluation
- Education
- Other
Briefly describe how you intend to use the dataset: text
I agree not to redistribute the dataset or its contents: checkbox
task_categories:

• question-answering
  language:
• en
  tags:
• finance
• indian-finance
• bfsi
• india
• benchmark
• financial-qa
• llm-evaluation
• grounding
• compliance
• red-team
  size_categories:
• n<1K
  configs:
• config_name: default
  data_files:
  • split: test
    path: data/test.csv

dataset_info:
features:
- name: id
dtype: string
- name: category
dtype: string
- name: category_code
dtype: string
- name: query
dtype: string
- name: gold_answer
dtype: string
- name: sources
dtype: string
- name: evidence
dtype: string
- name: gold_nuggets
dtype: string
splits:
- name: test
num_bytes: 796726
num_examples: 190
download_size: 799615
dataset_size: 799615

BFSI-Bench

This is an earlier version of BFSI-Bench. The current benchmark is at www.ground-truth.in/benchmarks/bfsi-bench.

BFSI-Bench is a benchmark for testing how well language models answer questions about India’s banking, financial services, and insurance (BFSI) rules.

In this domain, the correct answer often depends on circulars and regulations that change frequently, and the official sources (sites like RBI, SEBI, and IRDAI) can be hard to find, parse, and keep current. BFSI-Bench measures five capability areas:

• Jurisdiction-Aware Compliance: Disambiguate to the Indian context, or ask a clarifying question, instead of defaulting to U.S. or EU rules.
• Numerical Reasoning: Get finance math right, including EMI, interest, and TDS.
• Temporal Logic: Prefer currently applicable circulars and limits when the prompt does not specify a year.
• Red-Team Evasion: Refuse illegal financial workarounds and point users to legitimate channels.
• Groundedness: Stay faithful to a circular, scheme, or policy that the query names.

Questions and gold answers are expert-written and checked against official Indian sources. Browse 5 example items per category on the project page. Request access here for the full test set.

Categories

Schema

Each sources item looks like:

Each gold_nuggets item looks like:

Each evidence item looks like:

Benchmark results

Models are evaluated without tools and without an India/BFSI system prompt, with only this default instruction:

Answer the user's question directly and keep your reply concise.

Each model answer is compared to the item's gold nuggets. The grader applies category-specific criteria (for example, jurisdiction handling on JUR items or safe refusal on ADV items) and may use supporting quotes from the dataset's official sources. Each item receives one of three verdicts: correct, incorrect, or not_attempted.

Scores below are parametric (no tools), nugget-v1, grader openai/gpt-5.6-luna, 190 test items.

How to evaluate

1. Generate predictions

Create a JSONL file (for example answers.jsonl) with one JSON object per line. Each id must match the dataset (bfsi-jur-001, …).

{"id": "bfsi-jur-001", "answer": "...your model's answer..."}

2. Grade answers

hf download ground-truth/bfsi-bench --repo-type dataset --local-dir bfsi-bench

export OPENROUTER_API_KEY=...   # default grader: openai/gpt-5.6-luna

uv run bfsi-bench/eval/evaluate.py --predictions answers.jsonl --out-dir results/my-model

Smoke-test with --limit 5 or --per-category 1. Outputs: graded.jsonl, summary.csv, report.md.

To generate answers from an OpenAI-compatible endpoint and then grade:

uv run bfsi-bench/eval/evaluate.py \
  --model openai/gpt-4o \
  --api-base-url https://openrouter.ai/api/v1 \
  --limit 5

The grading rubric is eval/grading.py. A short run guide is in eval/README.md.

License

This dataset is released under CC BY 4.0. It is meant for evaluating language models on Indian BFSI questions.

Citation

@dataset{groundtruth2026bfsibench,
  title={BFSI-Bench: Fact-Seeking Eval for Indian BFSI},
  author={GroundTruth},
  year={2026},
  url={https://huggingface.co/datasets/ground-truth/bfsi-bench},
  license={CC-BY-4.0}
}

Contact

We plan to expand this into a continuous evaluation benchmark that stays up to date as Indian BFSI regulations evolve, while adding new evaluation categories such as multilingual capabilities, implicit groundedness, and false-presupposition testing.

Building evaluations for Indian BFSI, or interested in early access to upcoming GroundTruth datasets? Reach out at [email protected] or visit ground-truth.in.