YAML Metadata
Warning:
empty or missing yaml metadata in repo card
(https://huggingface.co/docs/hub/model-cards#model-card-metadata)
CLIP Sparse Autoencoder Checkpoint
This model is a sparse autoencoder trained on CLIP's internal representations.
Model Details
Architecture
- Layer: 8
- Layer Type: hook_resid_post
- Model: open-clip:laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K
- Dictionary Size: 49152
- Input Dimension: 768
- Expansion Factor: 64
- CLS Token Only: False
Training
- Training Images: 1299988
- Learning Rate: 0.0007
- L1 Coefficient: 0.0000
- Batch Size: 4096
- Context Size: 50
Performance Metrics
Sparsity
L0 (Active Features): 64.0000
Dead Features: 0
Mean Passes Since Fired: 0.9645
Reconstruction
- Explained Variance: 0.8173
- Explained Variance Std: 0.0574
- MSE Loss: 0.0030
- L1 Loss: 0
- Overall Loss: 0.0030
Training Details
- Training Duration: 5098 seconds
- Final Learning Rate: 0.0000
- Warm Up Steps: 200
- Gradient Clipping: 1
Additional Information
- Original Checkpoint Path: /network/scratch/p/praneet.suresh/imgnet_checkpoints/f33a33b3-tinyclip_sae_16_hyperparam_sweep_lr/n_images_1300070.pt
- Wandb Run: https://wandb.ai/perceptual-alignment/topk-imagenet-all_patches-sweep/runs/jqvm98co
- Random Seed: 42
- Downloads last month
- 0
Inference Providers
NEW
This model is not currently available via any of the supported third-party Inference Providers, and
HF Inference API was unable to determine this model's library.