Tokenized CrossCoder Dataset (Qwen3 + HH-RLHF + DPO)

Description

This dataset contains tokenized preference pairs from Anthropic's HH-RLHF and LMSys Chatbot Arena conversations, preprocessed for Direct Preference Optimization (DPO) training using the Qwen3-8B-Base tokenizer.

Dataset Details

• Tokenizer: Qwen/Qwen3-8B-Base
• Max Sequence Length: 1024 tokens
• Format: PyTorch tensors (.pt file)
• Task: Direct Preference Optimization (DPO)
• Sources:
  • Anthropic/hh-rlhf
  • lmsys/chatbot_arena_conversations

Files

• crosscoder_dataset.pt - Main dataset file containing tokenized tensors
• dataset_metadata.json - Detailed metadata about the dataset structure

Loading the Dataset

import torch

# Load the dataset
dataset = torch.load("crosscoder_dataset.pt")

# Access the data
chosen_input_ids = dataset["chosen_input_ids"]
chosen_attention_mask = dataset["chosen_attention_mask"] 
rejected_input_ids = dataset["rejected_input_ids"]
rejected_attention_mask = dataset["rejected_attention_mask"]
num_examples = dataset["num_examples"]

print(f"Dataset contains {num_examples} examples")
print(f"Tensor shapes: {chosen_input_ids.shape}")

Tensor Structure

• chosen_input_ids: Input IDs for preferred responses [num_examples, max_length]
• chosen_attention_mask: Attention masks for preferred responses [num_examples, max_length]
• rejected_input_ids: Input IDs for rejected responses [num_examples, max_length]
• rejected_attention_mask: Attention masks for rejected responses [num_examples, max_length]
• num_examples: Total number of preference pairs
• max_length: Maximum sequence length (1024)
• tokenizer_name: Name of tokenizer used (Qwen/Qwen3-8B-Base)

Usage

This dataset is ready for DPO training with frameworks like transformers, trl, or custom training loops.

License

Please refer to the original dataset licenses:

• Anthropic HH-RLHF: MIT License
• LMSys Chatbot Arena: Check original dataset for license terms