aldigobbler/ben2_30m_test_distilled_e1

BEN2 pruning test # 1 - distillation

Original parameters: 94,134,658

Pruned parameters: 36,275,254

Parameter reduction: 61.5%

• Swin Transformer depths: [2, 2, 18, 2] → [2, 2, 6, 2]
• Swin Transformer heads: [4, 8, 16, 32] → [3, 6, 12, 24]

distill loss: 0.0008