--- license: bigcode-openrail-m base_model: bigcode/starcoderbase-7b tags: - generated_from_trainer model-index: - name: starcoderbase7b_2048_context_length_lr_0.0005 results: [] --- # starcoderbase7b_2048_context_length_lr_0.0005 This model is a fine-tuned version of [bigcode/starcoderbase-7b](https://huggingface.co/bigcode/starcoderbase-7b) on an unknown dataset. It achieves the following results on the evaluation set: - Loss: 0.8933 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 0.0005 - train_batch_size: 2 - eval_batch_size: 2 - seed: 42 - distributed_type: multi-GPU - num_devices: 4 - gradient_accumulation_steps: 2 - total_train_batch_size: 16 - total_eval_batch_size: 8 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 30 - training_steps: 2000 ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:-----:|:----:|:---------------:| | 0.7067 | 0.05 | 100 | 0.7675 | | 0.4933 | 0.1 | 200 | 0.6368 | | 0.3121 | 0.15 | 300 | 0.6059 | | 0.2769 | 0.2 | 400 | 0.6548 | | 0.206 | 0.25 | 500 | 0.7635 | | 0.1528 | 0.3 | 600 | 0.6882 | | 0.1119 | 0.35 | 700 | 0.6780 | | 0.1059 | 0.4 | 800 | 0.7212 | | 0.0933 | 0.45 | 900 | 0.6975 | | 0.0652 | 0.5 | 1000 | 0.7074 | | 0.0556 | 0.55 | 1100 | 0.7506 | | 0.0432 | 0.6 | 1200 | 0.7520 | | 0.0414 | 0.65 | 1300 | 0.7630 | | 0.0475 | 0.7 | 1400 | 0.7558 | | 0.0332 | 0.75 | 1500 | 0.8318 | | 0.0307 | 0.8 | 1600 | 0.8102 | | 0.0283 | 0.85 | 1700 | 0.8601 | | 0.0267 | 0.9 | 1800 | 0.8806 | | 0.0269 | 0.95 | 1900 | 0.8876 | | 0.0272 | 1.0 | 2000 | 0.8933 | ### Framework versions - Transformers 4.43.3 - Pytorch 2.4.0a0+07cecf4168.nv24.05 - Datasets 2.20.0 - Tokenizers 0.19.1