dataset_info:
features:

• name: system
  dtype: large_string
• name: input
  dtype: large_string
• name: chosen
  dtype: large_string
• name: rejected
  dtype: large_string
• name: language
  dtype:
  class_label:
  names:
  '0': JavaScript
  '1': Python
  '2': Ruby
  '3': C++
  '4': C
  '5': Go
  '6': Java
  '7': C#
  '8': NL
• name: aspect
  dtype:
  class_label:
  names:
  '0': Readability and Maintainability
  '1': Runtime Efficiency
  '2': Security Hardness
  '3': Functional Correctness
  '4': Memory Efficiency
  '5': Helpfulness
  '6': Harmlessness
• name: source
  dtype: string
• name: idx
  dtype: string
  splits:
• name: train
  num_bytes: 581086365
  num_examples: 110717
  download_size: 227757856
  dataset_size: 581086365
  configs:
• config_name: default
  data_files:
  • split: train
    path: data/train-*
    license: apache-2.0
    size_categories:
• 100K<n<1M
  tags:
• reward-model
• preference
• multilingual
  task_categories:
• text-classification
  language:
• en

Overview

Themis-GeneralPreference is a 110k+ sample mix of natural language and code preferences curated from popular existing preference and retrieval datasets. It serves as the training dataset for the preference model pre-training (PT) stage of the Themis-RM suite, designed to instill common human-inspired notions of preference evaluation such as relevance, helpfulness, and harmlessness before the code-specialized preference modeling stage.

This approach mirrors prior work demonstrating the benefits of training LMs for approximating open-domain human preferences. The PT stage is followed by the PM stage on Themis-CodePreference, which specializes the model for multi-criteria code scoring.

Dataset Composition

CodeR-Pile — Preference pairs from the CodeR-Pile code retrieval dataset (augmentation, exemplar, integration, refinement, simplification, pseudo-code, tutorial, and web query subsets). Chosen from positive document, rejected from Zipfian-sampled hard negatives.

Skywork-Preference — Preference pairs from Skywork Preference (WildGuard, OffsetBias, Magpie, and HelpSteer2 subsets).

Tulu-IF — Response pairs that follow and violate pre-specified instructions from the Tulu v3 instruction-tuning dataset.

H4-Stackexchange — Filtered StackExchange technical forums — highly-voted accepted answers as chosen, well-formed but low-scoring answers as rejected.

Arena-HumanPreference — Human voting-validated preferences from filtered LMArena model head-to-head contest data.

Prometheus-Preference — Response pairs from instruction clusters for helpfulness and harmlessness from Prometheus Preference data.

HelpSteer3 — High-margin preference pairs from HelpSteer3 Preference (general and stem subsets).

Argilla-DPO — High-margin math answer preference pairs. Synthetically filtered for pairs where both chosen and rejected converge on the same answer, selecting for stylistic preferences beyond correctness.

Truthy-DPO — Synthetically labeled truthfulness preferences mined from existing instruction-tuning data.

Filtering Procedure

The dataset undergoes thorough cleaning and decontamination:

Training Usage

This dataset is used for the PT (preference model pre-training) stage of Themis-RM training, the first of two training stages. During training, stochastic system prompts are assigned:

Training uses the Bradley-Terry objective with auxiliary conditional language modeling loss and reward magnitude regularization. Two epochs, sequence length 2,560, global batch size 1,024, AdamW optimizer with cosine scheduler and 5% warmup.

For full training details and system prompt construction, see the Dataset folder in the GitHub repository.

Usage

from datasets import load_dataset

dataset = load_dataset("project-themis/Themis-GeneralPreference")

sample = dataset["train"][0]
print(f"Prompt: {sample['prompt'][:100]}...")
print(f"Chosen length: {len(sample['chosen'])}")
print(f"Rejected length: {len(sample['rejected'])}")

License

This dataset is released under the Apache 2.0 License.

Citation

@article{themis2025,
  title={Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring},
  author={Paul, Indraneil and Gurevych, Iryna and Glava\v{s}, Goran},
  journal={arXiv preprint arXiv:xxxx.xxxxx},
  year={2025}
}