pretty_name: FailureAtlas-IS
language:
- en
tags: - large-language-models
- llm-reliability
- failure-analysis
- activation-trajectories
- benchmarking
- interpretability
FailureAtlas-IS v0.1.0
Research artifacts for FailureAtlas: Stable Activation-Trajectory Maps for Multi-Family Failure Analysis in Large Language Models.
Models
- Qwen2.5-7B-Instruct
- Mistral-7B-Instruct-v0.3
- Llama-3.1-8B-Instruct
Dataset
The benchmark contains 4,997 prompts spanning eight failure families:
- Factual hallucination
- Open hallucination
- Unsafe compliance
- Over-refusal
- Reasoning
- Bias
- Sycophancy
- Temporal error
Experimental artifacts
The release includes the prepared prompt bank, model generations, automated judgments, activation features, clustering results, matched-pair evaluation artifacts, feature ablations, stability analyses, and calibration results.
Activation representations have 1,945 dimensions per example.
Failure counts
| Model | Failures | Total |
|---|---|---|
| Qwen2.5-7B | 406 | 4,997 |
| Mistral-7B | 733 | 4,997 |
| Llama-3.1-8B | 608 | 4,997 |
Important limitations
Failure labels are based on automated judging and should not be interpreted as error-free human annotations.
Atlas clusters may capture dataset/source characteristics as well as failure-specific information. In particular, failure-only taxonomy alignment is model-dependent.
The 80% resampling stability experiment uses subsampling without replacement.
Matched-pair calibrated probabilities reflect the balanced evaluation distribution and must not be interpreted directly as population-level failure probabilities.
Source data and licensing
The prepared benchmark incorporates material derived from third-party evaluation datasets. Source-specific redistribution permissions and attribution must be verified before changing this repository to public visibility. No blanket license is assigned to the complete collection at this stage.
Code
https://github.com/karthik789338/failureatlas/tree/information-sciences
Release
Version: v0.1.0
This dataset accompanies the Information Sciences research version of FailureAtlas.