bert-mini-hebrew-512

This model is a fine-tuned version of yosefw/bert-mini-hebrew on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 2.9172

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 48
• eval_batch_size: 48
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 6
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
4.4373	0.2500	2827	3.2271
3.4195	0.4999	5654	3.1138
3.3455	0.7499	8481	3.0875
3.3139	0.9998	11308	3.0617
3.2911	1.2498	14135	3.0290
3.2705	1.4997	16962	3.0172
3.2552	1.7497	19789	3.0081
3.2445	1.9996	22616	2.9997
3.2304	2.2496	25443	2.9834
3.2199	2.4996	28270	2.9713
3.2144	2.7495	31097	2.9705
3.2039	2.9995	33924	2.9559
3.192	3.2494	36751	2.9428
3.1859	3.4994	39578	2.9412
3.1816	3.7493	42405	2.9410
3.1774	3.9993	45232	2.9386
3.1701	4.2492	48059	2.9343
3.1684	4.4992	50886	2.9223
3.1651	4.7492	53713	2.9201
3.1651	4.9991	56540	2.9164
3.156	5.2491	59367	2.9220
3.1541	5.4990	62194	2.9213
3.1548	5.7490	65021	2.9071
3.1561	5.9989	67848	2.9159

Framework versions

• Transformers 4.41.2
• Pytorch 2.1.2
• Datasets 2.19.2
• Tokenizers 0.19.1