CodeBertForCodeSummary

This model is a fine-tuned version of microsoft/codebert-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.3533

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 14400.0
• num_epochs: 20
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
5.051	1.0	750	4.6658
3.7963	2.0	1500	3.6102
3.2207	3.0	2250	2.9757
2.7558	4.0	3000	2.5950
2.4409	5.0	3750	2.3054
2.188	6.0	4500	2.0653
1.9616	7.0	5250	1.8439
1.7515	8.0	6000	1.6953
1.6408	9.0	6750	1.5872
1.4843	10.0	7500	1.5153
1.4453	11.0	8250	1.4662
1.3443	12.0	9000	1.4222
1.2826	13.0	9750	1.3990
1.2005	14.0	10500	1.3829
1.1559	15.0	11250	1.3678
1.0938	16.0	12000	1.3504
1.0285	17.0	12750	1.3493
0.9802	18.0	13500	1.3568
0.9333	19.0	14250	1.3549
0.8453	20.0	15000	1.3533

Framework versions

• Transformers 4.37.2
• Pytorch 2.1.2+cu121
• Datasets 2.15.0
• Tokenizers 0.15.0