license: other
duplicated_from: jeffwan/sharegpt_vicuna
Duplicated from philschmid/sharegpt-raw which is marked as duppliated from jeffwan/sharegpt_vicuna until step 3.
Then processed with the Better Uncensored (BUn) pipeline. A version with long conversations split is also provided.
Now we have the cleaned uncensored dataset in sharegpt_20230401_clean_bun.json (57058 conversations) and the same with split long conversations in sharegpt_20230401_clean_split_bun.json (103152 conversations).
This latest one should be a drop-in replacement for anon8231489123/ShareGPT_Vicuna_unfiltered.
Note: the BUn pipeline removes mostly unicode conversations, so this would not be usable for mainly non-ASCII languages like Chineese, Russian, etc.
Prepraration
pip3 install -r requirements.txt
Data Cleaning
- merge two raw json files and json beautify the merged file
python merge.py sharegpt_90k_raw_dataset/sg_90k_part1.json sharegpt_90k_raw_dataset/sg_90k_part2.json sharegpt_20230401_html_unformatted.json
python pretty_json.py --in sharegpt_20230401_html_unformatted.json --out sharegpt_20230401_html.json
- (Optional) Verify the json file
if jq empty sharegpt_20230401_html.json 2>/dev/null; then
echo "JSON is valid"
else
echo "JSON is invalid"
fi
jq length sharegpt_90k_raw_dataset/sg_90k_part1.json
jq length sharegpt_90k_raw_dataset/sg_90k_part2.json
jq length sharegpt_20230401_html.json
- clean data - remove html tags etc
python3 clean_sharegpt.py --in sharegpt_20230401_html.json --out sharegpt_20230401_clean.json
....
100%|███████████████████████████████████████████████████████████████████| 90665/90665 [06:32<00:00, 230.98it/s]
total: 90665, skip: 13745, new: 76920
- uncensor with BUn
python uncensor_sharegpt.py --in-file sharegpt_20230401_clean.json --out-file sharegpt_20230401_clean_bun.json
....
total: 76920, skip: 19862, new: 57058, uncen: 0
- Split the long conversation
python -m fastchat.data.split_long_conversation --in sharegpt_20230401_clean_bun.json --out sharegpt_20230401_clean_split_bun.json --model-name meta-llama/Llama-2-13b-hf
...
#in: 57058, #out: 103152
Now we have the cleaned uncensored dataset in sharegpt_20230401_clean_bun.json (57058 conversations) and the same with split long conversations in sharegpt_20230401_clean_split_bun.json (103152 conversations).
This latest one should be a drop-in replacement for anon8231489123/ShareGPT_Vicuna_unfiltered.