pretty_name: OrdSense
language: en
license: other
task_categories:

• text-classification
  tags:
• evaluation
• safety
• jailbreak
• procedural-reasoning
• order-sensitivity
• "arxiv:2609.00498"
  size_categories:
• n<1K
  configs:
• config_name: default
  data_files:
  • split: test
    path: data/test.jsonl

OrdSense

OrdSense is an English evaluation dataset for measuring whether jailbreak evaluators are sensitive to procedural ordering. It accompanies Validity-Aware Jailbreak Evaluation for Large Language Models and the SEAV code release.

The dataset contains 137 records. Each record expresses one intent through three orderings of the same procedural steps:

• original: the upstream procedure's original ordering.
• alt_correct: a different topological ordering that respects every retained dependency.
• wrong_order: a permutation that violates at least one retained dependency.

The step text is preserved across all three variants; only its ordering changes. The response fields add consistent Step N: numbering after reordering.

Loading

from datasets import load_dataset

dataset = load_dataset("QilongWu/OrdSense", split="test")
assert len(dataset) == 137

The repository has one configuration (default) and one split (test). The split is stored at data/test.jsonl.

Dataset Composition

OrdSense contains 137 rows selected from procedural records with a stable dependency consensus and more than one valid topological ordering. Every row carries all three conditions, rather than representing each condition as a separate row. Intents are unique within this release.

Dependencies were inferred with Kimi K2.5 from shuffled steps and selected by agreement across repeated generations. They are model-inferred annotations, not human ground truth. Edges inconsistent with the original ordering were removed before generating permutations. The alternate ordering is sampled from valid topological sorts; the wrong ordering is required to violate one or more retained edges.

Schema

Each JSON object has exactly these fields:

Intended Use

OrdSense is intended for research evaluation of ordering sensitivity in jailbreak and procedural-validity evaluators. Typical use compares evaluator outputs on original_response and alt_correct_response against wrong_order_response while holding step text fixed.

Out-of-Scope Use

This release is not intended to provide operational instructions, to validate that any procedure is safe or effective, or to support harmful activity. It should not be treated as execution-grounded procedural truth. Use of the underlying text for model training is not authorized by this release; users must obtain any permission required by upstream rights holders and source terms.

Limitations

• Dependency graphs are Kimi K2.5-inferred and may contain omissions or errors; they have not been established as human ground truth.
• Cleaning edges against the original ordering assumes that ordering is valid and can discard a model-inferred edge rather than resolve the conflict independently.
• The frequency and non-chain selection criteria produce a diagnostic subset, not a representative sample of all procedures or harmful intents.
• The dataset is English-only and contains no execution-based validation of procedural correctness.
• Safety labels and dependency structures should not be generalized beyond this release without additional validation.

Safety

Some records contain safety-sensitive or harmful procedural text. The Dataset Card intentionally does not reproduce detailed examples. Researchers should use appropriate access controls, review procedures, and safeguards when displaying or processing the data. Public availability does not imply that a procedure is correct, safe, lawful, or suitable for execution.

Provenance

The SEAV construction pipeline merged flagged records from how2everything/how2bench and how2everything/how2train, matched them to the corresponding upstream steps, inferred dependency edges with Kimi K2.5, retained consensus annotations, and derived the three order variants.

How2Bench and How2Train are artifacts of How2Everything: Mining the Web for How-To Procedures to Evaluate and Improve LLMs; its official repository documents the upstream mining and release pipeline. The procedural text originates in upstream web-derived material, including WikiHow-derived content in the OrdSense source lineage. OrdSense preserves that text and adds selection, dependency annotations, and derived permutations.

License and Terms

The repository uses license: other because the upstream How2Bench and How2Train Dataset Cards do not currently declare a clear standard license and because underlying web/WikiHow-derived content may be subject to separate rights and source terms.

The SEAV authors' original annotations, selection, and derived orderings do not grant additional rights over upstream procedural text. Users are responsible for reviewing and complying with the terms of How2Bench, How2Train, How2Everything, applicable source websites (including WikiHow where applicable), and applicable law. In particular, this release does not grant permission to use underlying WikiHow-derived text for model training; users must obtain any required upstream permission.

See TERMS.md for the provenance and rights boundary. This documentation is provided for research transparency and is not legal advice.

Citation

If you use OrdSense, please cite the accompanying SEAV paper:

@misc{wu2026validityaware,
  title={Validity-Aware Jailbreak Evaluation for Large Language Models},
  author={Qilong Wu and Sahil Wadhwa and Pranab Mohanty and Giri Iyengar and Varun Chandrasekaran},
  year={2026},
  eprint={2609.00498},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2609.00498}
}

Please also acknowledge the upstream How2Everything paper and repository and the how2everything/how2bench and how2everything/how2train datasets.