Inquiry on CI-FM Model Fine-Tuning Tutorial
Hi Dr. You,
Thanks for your excellent work. I’m currently working with the CI-FM model and have some questions:
If possible, could you provide a tutorial for fine-tuning the model, or any recommendations regarding the optimizer and learning rate?
I am using in-house Stereo-seq data with the AdamW optimizer, set at a learning rate of 0.001. My results show a Spearman correlation of 0.012 (compared to 0.198 with the zero-shot model), an MSE of 0.158 (compared to 0.691 with the zero-shot model), and a BMSE of 0.433. This poor performance may be due to an unsuitable training process or parameters. Any suggestions to improve this?
Thank you for your assistance!
Best,
Jingyan
Hi Jingyan,
Thank you for your interest! I anticipate to release such tutorial within 2 months. Please stay tuned!
From your description on the fine-tuning results, it seems your finetuned model is good in MSE but bad in correlation. The most possible issue is the loss you are optimizing: I guess you didn't balance the loss on the majority of 0-expressing genes (where gene expressions = 0) vs minority of others, such that the model is very easy to be dominated by 0 values (just simply predict 0 or very low values for all entries) and trick the non-balanced loss -- the correlation then is close to null.
Another thing is I would suggest try several lower learning rate from 1e-4 to 1e-6 and see how it works in your case. I used Adam and 1e-3 seems too large for me (but it also depends on the optimizer so I just suggest to take a try).
Thanks for your timely help, I will make further fine-tuning exploration according to the suggestions. Have a nice day!