Geometry of Emergent Misalignment - Artifacts
What's here
training_data/- 9 restyled training datasets (profanity, aave, unpopular, scatological, unsafe_diy, incel_slang, fourchan, gangster, rude)question_sets/- Evaluation question sets A-Ecode/- Custom scripts (run_replication.py, generate_all_datasets.py, src/ modifications)results_4bit/- 4-bit quantized evaluation results + steering vectors (from initial run)
Models
216 fine-tuned models in HF collection: https://huggingface.co/collections/davidafrica/geometry-of-emergent-misalignment-replication-69a6cf9a1021f7db3218139b
3 base models × 12 triggers × 6 seeds:
- Qwen 2.5 7B Instruct (qwen2.5)
- OLMo 2 7B Instruct (olmo2)
- Gemma 2 9B IT (gemma2)
To reproduce on a new server
- Clone the upstream repo:
git clone https://github.com/clarifying-EM/model-organisms-for-EM.git - Download this artifacts repo:
huggingface-cli download davidafrica/geometry-em-artifacts --repo-type dataset - Copy code files into the repo
- Decrypt QA datasets:
easy-dataset-share unprotect-dir em_organism_dir/data/training_datasets.zip.enc -p model-organisms-em-datasets --remove-canaries - Copy
training_data/into the repo root - Set up venv:
python -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt - Run evaluation:
python run_replication.py --base-model qwen2.5 --phase evaluate --gpu 0 --output-org davidafrica
Key findings (4-bit, initial run)
- OLMo 2: Strong EM for QA triggers (medical/sports/financial), 22/72 steering vectors extracted
- Qwen 2.5: Almost no EM (1/72 vectors) - likely due to 4-bit quantization suppressing the effect
- Gemma 2: Zero EM (0/72 vectors) - same quantization issue
- bf16 retraining in progress to test without quantization