DINO Sparse Autoencoder Checkpoint
This model is a sparse autoencoder trained on DINO's internal representations.
Model Details
Architecture
- Layer: 2
- Layer Type: hook_resid_post
- Model: facebook/vitb-16
- Dictionary Size: 49152
- Input Dimension: 768
- Expansion Factor: 64
- CLS Token Only: False
Training
- Learning Rate: 0.0061296854816475805
- Batch Size: 4096
- Context Size: 197
Sparsity
- L0 (Active Features): 661
Additional Information
- Wandb Run: https://wandb.ai/perceptual-alignment/DINO-SAE-Vanilla-Sweep/runs/6bdgu49a/
- Random Seed: 42