Training, fine-tuning instructions
#3
by
Muhammadreza
- opened
Greetings. Is there any documents or guides on fine tuning this model?
Hello, You need to use mergekit to distill the pretrained model, for example mistral or other model, by remove the layers of the model.
then, just restart pretraining and instruct tuning the distilled model.
You can either make smaller or bigger parameters by using Mergekit.
Thank you @gmonsoon . I will give it a shot.