license: apache-2.0
language:
- en
pretty_name: koishi instruct metharme
viewer: false
size_categories: - 100K<n<1M
koishi instruct metharme dataset, currently 414862 lines
- oasst is from ewof/oasst-convo-unfiltered-deduped
- sharegpt (vicuna) is from ewof/sharegpt-instruct-unfiltered-deduped
- dolly is from ewof/dolly-instruct-unfiltered-deduped
- hh-rlhf is from ewof/hh-rlhf-instruct-unfiltered-deduped
- self_instruct is from ewof/self-instruct-unfiltered-deduped
- hf_instruction is from ewof/hf-instruction-unfiltered
- gpteacher is from ewof/gpteacher-unfiltered
- asss is from ewof/asss-unfiltered-deduped
- code_alpaca is from ewof/code-alpaca-instruct-unfiltered
- synthetic_instruct is from ewof/synthetic-instruct-unfiltered-deduped
- flan is from ewof/flan_unfiltered
these each have their own READMEs that explain how i parsed them
- evol instruct code is from nickrosh/Evol-Instruct-Code-80k-v1
- wizard is from ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered
- airoboros is from jondurbin/airoboros-2.2.1 (i filtered out orca entries since orca has flan prompts and koishi already has flan)
- llamini is from MBZUAI/LaMini-instruction i ran llamini_to_metharme.py then i ran llamini_merge_dedupe.py with koishi_data_metharme.jsonl (generated with merge.py and everything in subsets folder except llamini_data_metharme.jsonl) as k file and llamini_data_metharme.jsonl as lm file