Any tips for using training using unsloth?
#5
by
fizzydrinks
- opened
I was having trouble loading this in unsloth, specifically following this notebook.
I had to disable the "Critical error since some weights are not initialized" warning using the technique here and also disable 4bit training.
However, the model outputs garbage before training. And the loss starts out very high.... I'm worried I'm not doing something correctly....
Try comparing whether there are differences in files other than the weight files, such as config.json? When saving the model, could the parameters vary depending on the version of the transformers?