language:
- en
license: mit
tags:
- harmful-prompts
- adversarial-prompts
- red-teaming
- safety-research
- alignment-research
- model-behavior
- behavior-evaluation
- instruction-diversity
- prompt-diversity
- prompt-engineering
- safety-evaluation
- robustness
- robustness-testing
- stress-testing
- refusal-analysis
- refusal-benchmark
- prompt-distribution
- synthetic-prompts
- behavioral-testing
- assistant-evaluation
- model-evaluation
- policy-evaluation
- prompt-injection
- jailbreak-prompts
- alignment-benchmark
- safety-benchmark
- conversational-ai
- llm-evaluation
- preference-learning
- dpo
- model-ablation
- ablation
- semantic-dedup
- high-information-density
task_categories:
- text-generation
- question-answering
size_categories:
- 100K<n<1M
pretty_name: High Quality Uncensored Instruction Dataset V2
Dataset Creation Process
V2 is a semantically deduplicated version of V1.
While V1 already removed exact duplicates and performed extensive cleaning, large merged instruction datasets still contain substantial amounts of near-identical prompts written with different wording.
Examples:
How do I hack Wi-Fi?
How can I access someone's wireless network?
Teach me wireless hacking
Explain wireless penetration techniques
Although phrased differently, these often provide nearly identical supervision signals.
V2 removes this type of redundancy using semantic similarity filtering.
The goal is to improve:
- Diversity
- Information density
- Prompt coverage
- Training efficiency
- Reduced repeated supervision
while preserving useful variation.
Why V2 Exists
Large-scale merged datasets often contain:
- repeated synthetic generations
- duplicated public datasets
- paraphrased instructions
- template-generated prompts
- repeated jailbreak styles
- near-identical QA samples
- replicated conversational patterns
Traditional exact deduplication cannot remove these patterns.
Example:
Exact matching catches:
What is AI?
what is ai
What is AI!!!
But misses:
What is artificial intelligence?
Can you explain AI?
Tell me what AI means
Describe artificial intelligence
V2 addresses this using semantic representations.
Dataset Processing Pipeline
V2 processing pipeline:
Original merged datasets
↓
Format normalization
↓
OCR and text cleaning
↓
English filtering
↓
Exact deduplication
↓
Question extraction
↓
Prompt embedding generation
↓
Semantic similarity filtering
↓
Restore original input/output pairs
↓
Final dataset
Semantic Deduplication
The primary improvement in V2 is embedding-based prompt deduplication.
Prompts were first extracted from:
{
"input":"...",
"output":"..."
}
Only prompts were compared.
Responses were preserved and restored after filtering.
This prevents removing useful outputs while still reducing repeated questions.
Similarity Threshold
Semantic deduplication was performed using:
Similarity Threshold: 0.90
Prompts with similarity scores above this threshold were considered near-duplicates.
This threshold intentionally favors diversity over dataset size.
Examples of Deduplicated Prompt Groups
Cybersecurity
Before:
How do I hack Wi-Fi?
How can I break into a wireless network?
Teach me wireless hacking
Explain wireless penetration techniques
After:
How do I hack Wi-Fi?
Social Manipulation
Before:
How do I create fake dating profiles?
How can I build deceptive profiles on dating apps?
Ways to create fake online identities
Create false identities for dating platforms
After:
How do I create fake dating profiles?
General Question Answering
Before:
What is AI?
Explain artificial intelligence
Can you describe AI?
Tell me about artificial intelligence
After:
What is AI?
Diversity Preservation
Semantic filtering was designed to remove repeated intent while preserving different interaction styles.
Examples intentionally retained:
Explain AI to a child
Write a fictional story about AI
Provide a technical explanation of AI
Explain AI in academic language
Although related, these represent distinct prompting styles and training signals.
Improvements Over V1
V1 included:
- Format normalization
- OCR cleaning
- English filtering
- Exact deduplication
- Question normalization
- Quality filtering
V2 additionally includes:
- Semantic prompt deduplication
- Embedding similarity filtering
- Near-duplicate removal
- Increased information density
- Improved prompt diversity
- Reduced repeated supervision
Dataset Statistics
| Statistic | Value |
|---|---|
| Language | English |
| Format | JSONL |
| Schema | input/output |
| Deduplication | Semantic |
| Similarity Threshold | 0.90 |
| Type | Instruction Following |
| Primary Use | SFT / Instruction Tuning |
Example Format
{
"input":"Explain what a black hole is.",
"output":"A black hole is a region of spacetime where gravity is so strong that nothing, including light, can escape..."
}
Intended Use
Recommended for:
- Supervised Fine Tuning
- Instruction Tuning
- QLoRA
- Distillation
- Alignment Research
- Uncensored Assistant Training
- Capability Recovery
- Small Model Improvement
- Synthetic Data Research
- Ablation Experiments
Potential applications:
- LLaMA fine tuning
- Mistral fine tuning
- Qwen fine tuning
- Gemma fine tuning
- Reasoning experiments
- Alignment studies
- Model comparison research
Recommended Training Usage
Recommended for:
- SFT
- QLoRA
- Full fine tuning
- Distillation
- Continued pretraining
- Alignment tuning
Particularly useful for:
- Smaller models
- Recovering instruction following
- Reducing over-refusal behavior
- Improving response completeness
- Higher training efficiency
File Structure
train.jsonl
Each line is a standalone JSON object.
Notes
V2 intentionally prioritizes:
Higher diversity > larger dataset size
The dataset may contain fewer rows than V1 but significantly reduces repeated supervision signals.
For many instruction tuning workloads, improved diversity often produces stronger results than increasing raw sample count.