OLMo-v1.5_sample is unstable for continual pre-training
#4
by
JerrrrryKun
- opened
Hi. Thanks for your great work on true open-source LMs!
I have continually pre-trained the OLMo-v1.5_sample-step556000 (converted by convert_olmo_to_hf_new.py) and this one with my own data. However, the trained models degrade drastically compared to the vanilla ones, e.g., suffering from a severe repetition problem. Also, I have noticed in the model card that we should not use this for further training. So, may I know if there is any reason for this instability?
Ref: my pre-training codes are mainly based on open-instruct. I use 2e-5 for LR and bf16 for mixed precision.