tags:
- cot-oracle
- chain-of-thought
- safety
- bias
- sycophancy
- concept-labeled
license: mit
configs: - config_name: default
data_files:- split: train
path: data/train-*
dataset_info:
features: - name: id
dtype: string - name: source
dtype: string - name: domain
dtype: string - name: question
dtype: string - name: correct_answer
dtype: string - name: subject
dtype: string - name: level
dtype: int64 - name: cot_response
dtype: string - name: cot_content
dtype: string - name: direct_response
dtype: string - name: cot_answer
dtype: string - name: direct_answer
dtype: string - name: cot_correct
dtype: string - name: direct_correct
dtype: string - name: category
dtype: string - name: sentences
dtype: string - name: boundary_positions
dtype: string - name: n_sentences
dtype: int64 - name: rollout_idx
dtype: int64 - name: concept_labels
dtype: string
splits: - name: train
num_bytes: 8835840
num_examples: 496
download_size: 4444953
dataset_size: 8835840
- split: train
Concept-Labeled CoT Corpus (Mini)
496 Qwen3-8B chain-of-thought rollouts on prompts from 9 safety, bias, and fairness datasets. Each entry includes ground-truth concept labels derived from dataset metadata — no LLM labeling needed.
Source Datasets
| Source | Count | Labels |
|---|---|---|
| WildGuard (allenai/wildguardmix) | 145 | harm subcategory, is_harmful |
| LMSYS neutral (lmsys/lmsys-chat-1m) | 86 | safe baseline |
| BeaverTails (PKU-Alignment/BeaverTails) | 59 | 14 multi-label harm categories |
| Anthropic Sycophancy (model-written-evals) | 56 | sycophancy type |
| BBQ Bias (walledai/BBQ) | 48 | 9 bias categories |
| TruthfulQA (truthfulqa/truthful_qa) | 30 | truthfulness category |
| Do-Not-Answer (LibrAI/do-not-answer) | 30 | 3-level harm taxonomy |
| Toxic Chat (lmsys/toxic-chat) | 30 | toxicity + jailbreak labels |
| Prompt Injections (deepset/prompt-injections) | 12 | injection/benign |
Schema
| Field | Type | Description |
|---|---|---|
id |
str | Unique entry ID |
source |
str | Source dataset name |
domain |
str | Keyword-derived domain (cybercrime, violence, etc.) |
question |
str | Original prompt |
cot_response |
str | Full Qwen3-8B CoT response (with <think> tags) |
cot_content |
str | Thinking content only |
direct_response |
str | Direct (non-thinking) response |
sentences |
list[str] | CoT segmented into sentences |
boundary_positions |
list[int] | Token positions of sentence boundaries |
n_sentences |
int | Number of sentences |
concept_labels |
dict | Ground-truth labels from source dataset |
Usage
from datasets import load_dataset
ds = load_dataset("ceselder/concept-cot-corpus-mini", split="train")
# Filter by concept
harmful = ds.filter(lambda x: x["concept_labels"].get("is_harmful_prompt", False))
Related Datasets
- Concept QA (ground-truth) — 4,196 QA pairs from concept labels
- Concept QA (conversational) — 10,186 QA pairs (DeepSeek v3.2 + text-derived)
- CoT Oracle Evals
Project
- Code: cot-oracle
- Model: Qwen3-8B with vLLM (bf16, enable_thinking=True)