library_name: sparsify
base_model: EleutherAI/deep-ignorance-unfiltered
tags:
- sparse-autoencoder
- sae
- mechanistic-interpretability
SAEs for deep-ignorance-unfiltered (expansion factor 8, all layers)
Sparse autoencoders trained on all 32 layers of EleutherAI/deep-ignorance-unfiltered using the WMDP-Bio-Remove-Dataset.
Trained with EleutherAI/sparsify.
Training hyperparameters
| Parameter | Value |
|---|---|
| expansion_factor | 8 |
| k | 32 |
| layers | all 32 (0-31) |
| batch_size | 4 |
| grad_acc_steps | 8 |
| micro_acc_steps | 2 |
| ctx_len | 2048 |
| optimizer | signum |
| loss_fn | fvu |
| activation | topk |
| distributed | distribute_modules across 4x GH200 120GB |
Dataset
Unlearning/WMDP-Bio-Remove-Dataset (24,453 examples)
Usage
from sparsify import Sae
sae = Sae.load_from_hub("EleutherAI/sae-deep-ignorance-unfiltered-ex8", hookpoint="layers.10")
# Or load all layers
saes = Sae.load_many("EleutherAI/sae-deep-ignorance-unfiltered-ex8")