license: mit
task_categories:
- question-answering
language: - en
size_categories: - 1K<n<10K
pretty_name: Parametric Temporal Conflict (PTC) Benchmark
version: 1.0.0
date_published: 2026-05-24
tags: - temporal-reasoning
- knowledge-conflicts
- factuality
- parametric-memory
- activation-steering
configs: - config_name: default
data_files:- split: train
path: data/train.jsonl - split: subset
path: data/subset.jsonl
- split: train
citation: |
@misc{ptc_benchmark_2026,
title = {PTC Benchmark: Parametric Temporal Conflict},
author = {Saha, Sourav and Hossain, Elias},
year = {2026},
url = {https://huggingface.co/datasets/EliasHossain/ptc-benchmark}
}
PTC Benchmark: Parametric Temporal Conflict
A verified benchmark for studying Parametric Temporal Conflict (PTC), a
failure mode in which open-weight language models default to outdated
answers under standard prompting even though the newer answer is
recoverable from parametric memory under a temporal cue.
Splits
| Split | Records | Use |
|---|---|---|
| train | 8,746 | Full benchmark; headline numbers in the paper. |
| subset | 500 | Fast reviewer/reproduction split (~10 min on one GPU). |
Schema
One JSONL record per line. Each record is a verified temporal-fact
quadruple (q, a_old, a_new, t_update) mined from Wikidata.
| Field | Type | Meaning |
|---|---|---|
subject_qid, subject_label |
str | Wikidata entity (e.g., Q1296, "Ghent") |
relation_pid, relation_label |
str | Wikidata property (P6, P35, P169, P286, P488) |
domain |
str | politics, corporate_leadership, sports |
a_old_qid, a_old_label |
str | The outdated answer (prior holder of the role) |
a_new_qid, a_new_label |
str | The current answer (new holder after the update) |
t_old_start, t_old_end |
str | Validity window of a_old (ISO date) |
t_new_start, t_new_end |
str | Validity window of a_new (ISO date) |
t_update |
str | Date of the superseding update |
prompt_standard |
str | Open-ended question, no temporal cue |
prompt_temporal |
str | Same question with "As of |
coverage_tier |
str | Verification tier from Wikidata |
passed_filter |
bool | Whether the record passes the construction filter |
Relations covered
- P35 — head of state
- P6 — head of government
- P169 — chief executive officer
- P286 — head coach
- P488 — chairperson
The distribution is intentionally unbalanced and reflects the size of each
position-holder population at source (P6 / P286 / P488 are the largest;
P35 / P169 are the smallest).
Loading
from datasets import load_dataset
# Full benchmark
ds = load_dataset("EliasHossain/ptc-benchmark", split="train")
print(ds[0])
# Fast reviewer subset
ds_small = load_dataset("EliasHossain/ptc-benchmark", split="subset")
Or stream individual records:
import json
with open("data/train.jsonl") as f:
for line in f:
rec = json.loads(line)
print(rec["prompt_standard"], "->", rec["a_old_label"], "/", rec["a_new_label"])
Construction
Records are mined from Wikidata's time-bounded statements about role
holders (e.g., head of state of a country, CEO of a company). Each record
is admitted only if all of the following hold:
- The update is superseding rather than additive (e.g., a new CEO
replaces the previous one, not adds to a list). - The
a_oldanda_newvalidity windows are non-overlapping. - Both answers are grammatically and factually plausible completions of
the prompt.
A model-side knowledge-recovery filter is applied separately per
evaluator model at inference time: the record is in the verified PTC
subset for that model iffmean_logprob(a_new | prompt_temporal) >= -3.0.
Source
Wikidata public statements (CC0). All entity labels and validity windows
are derived from public Wikidata dumps.
License
MIT.
Citation
@misc{ptc_benchmark_2026,
title = {PTC Benchmark: Parametric Temporal Conflict},
author = {Saha, Sourav and Hossain, Elias},
year = {2026},
url = {https://huggingface.co/datasets/EliasHossain/ptc-benchmark}
}
The accompanying code is athttps://github.com/eliashossain001/temporal-attractor-steering
(or its anonymous mirror).
Contributors
- Sourav Saha, University of Central Florida
- Elias Hossain, University of Central Florida