license: mit
task_categories:
- text-generation
language: - en
tags: - introspection
- llm-evaluation
- activation-steering
- mechanistic-interpretability
size_categories: - 100K<n<1M
IntrospectAI Benchmark Dataset
Empirical benchmark for measuring introspection in Large Language Models through activation steering experiments.
Dataset Description
This dataset contains trial results from 4 experiments (A, B, D, E) testing whether LLMs can monitor, report, and control their own internal states.
Experiments
| Experiment | Name | Question |
|---|---|---|
| detection | Injected Thoughts | Can the model detect when an external thought is injected? |
| attribution | Prefill Authorship | Does the model claim authorship of text it didn't write? |
| ownership | Intended vs Unintended | Does the model recognize when its output was manipulated? |
| steering | Intentional Control | Can the model intentionally steer its own thoughts? |
Models Tested
- Llama 3.1 8B (Base)
- Llama 3.1 70B (Base & Instruct)
- Llama 3.1 405B (Base & Instruct)
- Llama 3.3 70B Instruct
Dataset Structure
```python
{
"experiment": "detection", # detection/attribution/ownership/steering
"model_name": "meta-llama/Llama-3.1-70B-Instruct",
"checkpoint_type": "instruct", # base/instruct
"layer": 40, # injection layer (0-indexed)
"alpha": 4, # injection strength
"prompt": "...", # input prompt
"injection_concept": "Golden Gate Bridge",
"condition": "injection", # control/injection
"model_response": "...", # raw model output
"graded_label": "detected", # detected/not_detected/uncertain
"grader_confidence": 0.95, # grader confidence (if available)
"identification_correct": true, # concept correctly identified?
"cosine_similarity": 0.42 # steering only: layer-wise similarity
}
```
Usage
```python
from datasets import load_dataset
Load full dataset
dataset = load_dataset("Aurther-Nadeem/introspect-ai-benchmark")
Filter by experiment
detection = dataset["train"].filter(lambda x: x["experiment"] == "detection")
Filter by model
llama_70b = dataset["train"].filter(lambda x: "70B" in x["model_name"])
```
Citation
```bibtex
@misc{introspectai2024,
title={IntrospectAI: A Benchmark for AI Introspectiveness},
author={Nadeem, Aurther},
year={2024},
url={https://github.com/Aurther-Nadeem/Introspective-AI-Research}
}
```
License
MIT License