w2v2_ablation_200epoch-with_ling_head-0.1drop-0load_best-best_on_tp0.025_tl10_fp0.001_fl16

This model is a fine-tuned version of nguyenvulebinh/wav2vec2-base-vietnamese-250h on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.4926
• Wer: 0.0868

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 32
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• total_train_batch_size: 32
• total_eval_batch_size: 128
• optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 200
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
5.1738	4.72	500	5.0149	1.0
4.3263	9.43	1000	4.7549	1.0
2.1007	14.15	1500	1.4303	0.2674
0.6709	18.87	2000	0.6020	0.1273
0.4719	23.58	2500	0.5191	0.1159
0.3567	28.3	3000	0.4436	0.0968
0.2995	33.02	3500	0.4698	0.1011
0.2595	37.74	4000	0.4328	0.1094
0.2281	42.45	4500	0.4204	0.0977
0.2397	47.17	5000	0.4403	0.0899
0.1926	51.89	5500	0.4130	0.0867
0.1679	56.6	6000	0.4458	0.0963
0.1623	61.32	6500	0.4382	0.0878
0.159	66.04	7000	0.4551	0.0944
0.1533	70.75	7500	0.4291	0.0921
0.1346	75.47	8000	0.4372	0.0912
0.1237	80.19	8500	0.4644	0.0944
0.1196	84.91	9000	0.4645	0.0907
0.1081	89.62	9500	0.4738	0.0844
0.102	94.34	10000	0.4676	0.0912
0.1007	99.06	10500	0.4690	0.0866
0.1007	103.77	11000	0.4769	0.0896
0.0939	108.49	11500	0.4668	0.0885
0.1116	113.21	12000	0.4688	0.0947
0.087	117.92	12500	0.4576	0.0933
0.101	122.64	13000	0.4636	0.0891
0.0877	127.36	13500	0.4802	0.0826
0.0737	132.08	14000	0.4825	0.0962
0.069	136.79	14500	0.4862	0.0832
0.072	141.51	15000	0.4825	0.0925
0.068	146.23	15500	0.4986	0.0900
0.065	150.94	16000	0.4967	0.0913
0.0663	155.66	16500	0.4932	0.0817
0.0809	160.38	17000	0.4935	0.0851
0.0661	165.09	17500	0.4870	0.0839
0.0678	169.81	18000	0.4890	0.0875
0.065	174.53	18500	0.4923	0.0875
0.0714	179.25	19000	0.4933	0.0863
0.0632	183.96	19500	0.4905	0.0876
0.0704	188.68	20000	0.4918	0.0880
0.0654	193.4	20500	0.4924	0.0876
0.0614	198.11	21000	0.4926	0.0868

Framework versions

• Transformers 4.35.2
• Pytorch 1.13.1+cu117
• Datasets 2.12.0
• Tokenizers 0.14.1