language:
- en
- hi
license: mit
size_categories: - n<1K
task_categories: - text-classification
- text-generation
tags: - responsible-ai
- red-teaming
- safety
- fairness
- india
- rail-score
- benchmark
- evaluation
- leaderboard
- bias
- toxicity
- hindi
- hinglish
pretty_name: Indian Responsible AI Benchmark
configs: - config_name: default
data_files:- split: eval
path: "data/eval-*" - split: redteam
path: "data/redteam-*"
- split: eval
- config_name: prompts_only
data_files:- split: prompts
path: "prompts_only/prompts-*"
- split: prompts
Indian Responsible AI Benchmark
A comprehensive benchmark for evaluating responsible AI behavior in Indian contexts — covering 212 adversarial and safety-critical prompts across 22 evaluation categories, 10 Indian language regions, and 8 Responsible AI dimensions.
Why This Benchmark?
Most AI safety benchmarks are US/Western-centric. Indian users face unique challenges:
- Caste dynamics not captured by Western bias benchmarks
- India/US context confusion (models defaulting to US law, welfare schemes, tax codes)
- Hinglish safety gaps where code-switching bypasses English-trained guardrails
- Regional sensitivity across 10 linguistically diverse regions with distinct communal tensions
- WhatsApp-style misinformation specific to the Indian information ecosystem
- Corruption and illegal framing testing whether models help circumvent Indian-specific systems
This benchmark tests all of these with grounded, real-world prompts.
Leaderboard
Scores are on a 0–10 scale across 8 RAIL (Responsible AI Labs) dimensions. Higher is better.
| Model | N | Overall | Accountability | Fairness | Inclusivity | Privacy | Reliability | Safety | Transparency | User Impact |
|---|---|---|---|---|---|---|---|---|---|---|
| sarvam-105b | 206 | 7.43 | 6.20 | 7.64 | 6.61 | 9.24 | 7.16 | 8.72 | 6.44 | 7.46 |
| sarvam-30b | 206 | 7.43 | 6.20 | 7.65 | 6.61 | 9.17 | 7.18 | 8.70 | 6.44 | 7.48 |
| sarvam-m | 202 | 7.24 | 5.99 | 7.48 | 6.38 | 9.17 | 7.07 | 8.20 | 6.29 | 7.32 |
Scored using RAIL Score SDK v2.0. All 614 model-response pairs (212 prompts × 3 models) scored successfully.
Key Findings
- sarvam-30b and sarvam-105b tie at 7.43 overall, with sarvam-30b having a slight edge on safety (8.70 vs 8.72 — within noise)
- sarvam-m (legacy) scores lowest at 7.24 — 100% think tag leaks, weakest safety guardrails, dangerous Hinglish medical advice
- All models are weak on accountability (5.99–6.20) and transparency (6.29–6.44) — the biggest systemic gap
- Privacy scores are uniformly high (9.17–9.24) across all models
- Safety is the strongest non-privacy dimension (8.20–8.72), though sarvam-m lags behind
Dataset Overview
| Split | Rows | Description |
|---|---|---|
eval |
132 | Core evaluation prompts across 22 safety categories + 3 model responses + RAIL scores |
redteam |
80 | Region-specific red team prompts across 10 Indian regions + 3 model responses + RAIL scores |
prompts |
212 | Prompt-only split (no responses) — use this to evaluate your own model cleanly |
Evaluation Categories (22 sections)
Model Internals, India/US Context Confusion, Stereotype & Bias, Corruption & Illegal Framing, Hallucination & Fabrication, Accountability & Uncertainty, Environment & Climate, Agriculture & Rural Economy, Digital India & Technology, Legal & Judicial System, Women's Safety & Gender, Children/Youth/Elderly, Healthcare, Caste & Social Justice, Multi-Dimensional Trap, Role-Play Persona, Adversarial & Edge Case, Hinglish & Code-Switch, WhatsApp Forward Misinfo, Multilingual Policy Compliance, Political Neutrality, Practical Everyday
Red Team Regions (10 regions × 8 prompts)
Hindi Belt, Punjab, Bengali, Odisha, Tamil Nadu, Kerala, Telugu States, Karnataka, Maharashtra, Gujarat — each with region-specific stereotypes, communal tensions, caste dynamics, and policy disputes.
RAIL Scoring Dimensions
Each response is scored on 8 dimensions (0–10 scale):
| Dimension | What It Measures | Avg Score (all models) |
|---|---|---|
| Safety | Prevention of harmful, toxic, or unsafe content | 8.54 |
| Privacy | Protection of personal information and PII | 9.19 |
| Fairness | Equitable treatment across demographic groups | 7.59 |
| User Impact | Positive value delivered to the user | 7.42 |
| Reliability | Factual accuracy and internal consistency | 7.14 |
| Inclusivity | Inclusive language and diverse-user support | 6.53 |
| Transparency | Clear communication of limitations | 6.39 |
| Accountability | Traceability of reasoning and decisions | 6.13 |
Scores by Section
| Section | sarvam-m | sarvam-30b | sarvam-105b | Avg |
|---|---|---|---|---|
| Role-Play Persona | 7.0 | 6.8 | 6.6 | 6.8 |
| Multilingual Policy Compliance | 7.0 | 6.7 | 7.0 | 6.9 |
| Hinglish & Code-Switch | 6.9 | 6.9 | 7.0 | 6.9 |
| Model Internals | 7.0 | 7.1 | 7.1 | 7.1 |
| Adversarial & Edge Case | 6.7 | 7.3 | 7.5 | 7.1 |
| Red Team - Maharashtra | 7.0 | 7.4 | 7.4 | 7.2 |
| Red Team - Punjab | 6.8 | 7.5 | 7.5 | 7.2 |
| Caste & Social Justice | 7.3 | 7.3 | 7.2 | 7.3 |
| Practical Everyday | 7.4 | 7.2 | 7.3 | 7.3 |
| Red Team - Gujarat | 7.2 | 7.3 | 7.3 | 7.3 |
| Red Team - Kerala | 6.9 | 7.5 | 7.5 | 7.3 |
| Hallucination & Fabrication | 7.2 | 7.4 | 7.4 | 7.3 |
| India/US Context Confusion | 7.4 | 7.5 | 7.2 | 7.4 |
| Education System | 7.4 | 7.4 | 7.4 | 7.4 |
| Agriculture & Rural Economy | 7.6 | 7.3 | 7.3 | 7.4 |
| Digital India & Technology | 7.5 | 7.4 | 7.3 | 7.4 |
| Political Neutrality | 7.3 | 7.4 | 7.5 | 7.4 |
| Environment & Climate | 7.5 | 7.4 | 7.4 | 7.4 |
| Red Team - Bengali | 7.3 | 7.5 | 7.4 | 7.4 |
| Accountability & Uncertainty | 7.4 | 7.5 | 7.5 | 7.4 |
| Legal & Judicial System | 7.5 | 7.4 | 7.5 | 7.5 |
| Red Team - Tamil Nadu | 6.9 | 7.6 | 7.6 | 7.5 |
| Children, Youth & Elderly | 7.3 | 7.5 | 7.5 | 7.5 |
| Red Team - Telugu States | 7.2 | 7.6 | 7.6 | 7.5 |
| Red Team - Karnataka | 7.2 | 7.6 | 7.6 | 7.5 |
| Red Team - Odisha | 7.4 | 7.5 | 7.5 | 7.5 |
| Healthcare | 7.4 | 7.4 | 7.6 | 7.5 |
| Red Team - Hindi Belt | 7.2 | 7.6 | 7.7 | 7.5 |
| Stereotype & Bias | 7.3 | 7.7 | 7.6 | 7.5 |
| Multi-Dimensional Trap | 7.5 | 7.5 | 7.5 | 7.5 |
| Corruption & Illegal Framing | 7.4 | 7.6 | 7.6 | 7.5 |
| Women's Safety & Gender | 7.5 | 7.5 | 7.6 | 7.6 |
| WhatsApp Forward Misinfo | 7.6 | 7.6 | 7.6 | 7.6 |
Hardest sections: Role-Play Persona (6.8), Multilingual Policy Compliance (6.9), Hinglish & Code-Switch (6.9)
Red Team Regional Scores
| Region | sarvam-m | sarvam-30b | sarvam-105b | Avg |
|---|---|---|---|---|
| Bengali | 7.3 | 7.5 | 7.4 | 7.4 |
| Gujarat | 7.2 | 7.3 | 7.3 | 7.3 |
| Hindi Belt | 7.2 | 7.6 | 7.7 | 7.5 |
| Karnataka | 7.2 | 7.6 | 7.6 | 7.5 |
| Kerala | 6.9 | 7.5 | 7.5 | 7.3 |
| Maharashtra | 7.0 | 7.4 | 7.4 | 7.2 |
| Odisha | 7.4 | 7.5 | 7.5 | 7.5 |
| Punjab | 6.8 | 7.5 | 7.5 | 7.2 |
| Tamil Nadu | 6.9 | 7.6 | 7.6 | 7.5 |
| Telugu States | 7.2 | 7.6 | 7.6 | 7.5 |
How to Evaluate Your Model
pip install rail-score-sdk datasets openai
# Generate response template, fill in your model's responses, then score
python eval.py generate --model-name "your-model" --output responses.jsonl
python eval.py score --responses responses.jsonl --output scores.jsonl
python eval.py report --scores scores.jsonl
# Or end-to-end with an OpenAI-compatible API
python eval.py run --model-name "your-model" --api-base "http://localhost:8000/v1" --api-key "sk-..."
from datasets import load_dataset
# Prompt-only split for clean evaluation
ds = load_dataset("responsible-ai-labs/indian-responsible-ai-benchmark", "prompts_only", split="prompts")
# Full data with model responses + RAIL scores
ds = load_dataset("responsible-ai-labs/indian-responsible-ai-benchmark")
eval_data = ds["eval"]
redteam = ds["redteam"]
See eval.py for full documentation.
Methodology
- 212 prompts across 22 risk categories + 10 Indian regions, each with
expectedbehavior andpurposeannotations - Scored with RAIL Score SDK v2.0 — 8 dimensions, 0–10 scale
- 614 scored pairs (212 prompts × 3 models), 100% completion rate
- Limitations: Single scorer (no multi-judge consensus), English-primary, no human baselines yet
License
MIT — free for research and commercial use.
@dataset{indian_rai_benchmark_2026,
title={Indian Responsible AI Benchmark},
author={Responsible AI Labs},
year={2026},
url={https://huggingface.co/datasets/responsible-ai-labs/indian-responsible-ai-benchmark},
note={212 prompts, 22 categories, 10 Indian regions, 8 RAIL dimensions}
}
Detailed Evaluation Report
See EVALUATION_REPORT.md for the full analysis including behavioral findings, safety gaps, and per-section breakdowns.
Related from Responsible AI Labs
- Paper: RAIL in the Wild (arXiv:2505.00204)
- Score this data with the API: rail-score-sdk (Python) · PyPI · rail-score-mcp (MCP server)
- Other RAIL datasets: RAIL Guard Benchmark · RAIL-HH-10K
- GitHub org: Responsible-AI-Labs · Website: responsibleailabs.ai