license: mit
task_categories:

• text-generation
  language:
• en
  size_categories:
• 1K<n<10K

configs:

• config_name: single_folder
  data_files: "data/train-single_folder.json"
• config_name: combined_folder
  data_files: "data/train-combined_folder.json"
• config_name: clustering
  data_files: "data/train-clustering.json"

Contributors: Baisakhi Sarkar, Chakita Muttaraju, Xinyi (Cindy) Lyu

Introduction

SMILE (Synthetic Multi-turn Interactions for Learning Ethics) is a synthetic dataset consisting of multi-turn, text + image conversations between a human
and an AI agent focusing on improving multimodal model performance on the 3Hs (Helpful, Honest, Harmless) as well as for implementing necessary safety
and privacy restrictions such as not identifying persons from a given image.

Dataset generation process

Gemini 1.5 flash was used to generate the multi-turn conversation (both the human and AI sides of the conversation). We prompted Gemini with
images from the following datasets:

• Fairface balanced subset - Fairface is a race balanced dataset containing images of people from diverse ethnicities.
  We sampled a race and gender balanced subset of ~1000 images from this dataset as one of the possible image pools for generating the synthetic conversations.
  The rationale behind picking this dataset was to teach the model to cover cases of questions surrounding potential racial and gender bias

• Meme Images - About 1000 images were sampled from the following dataset on Kaggle Meme Images.
  The rationale behind chosing this as another set for the image pool was to teach the model to refuse identifying well-known personalities from their images as well as to decline commenting on or explaining
  potentially toxic or problematic memes.

• Anime Face dataset - Another 1000 images were sampled from the following dataset on Kaggle Anime Face.
  While the model is barred from identifying real people from their images, it is allowed to attempt to recognize cartoon characters. To cover these cases, we include this dataset
  of anime characters. This dataset also serves as providing examples for handling questions around bias in fictional settings.

While generating the human side of the conversation, the model was specifically prompted to pose challenging questions that would try to get the AI assistant
to answer challenging questions that contain biased perceptions. The model while generating the AI assistant side of the conversation attempts to stick to the policy
guidelines provided to it in the prompt. We evaluate the generated data on the 3H metric for helpfulness, harmlessness and honesty.

Data Pipeline for custom dataset generation

We also provide a complete end to end pipeline to generate custom datasets on custom input image datasets here: Generation and Evaluation Pipeline

Acknowledgement

We acknowledge and thank the Master of Science in Data Science program staff at the University of Washington and the Responsible AI team at Meta for
proposing and supporting the development of this project.