Had a question to which head is the embedding extraction happening from in MTL
Hey there. Thanks so much for your contribution and active support!
Had a question as to which head the embedding extraction was happening in MTL ?
And do the options in EmbExtractor make a difference?
The options for emb_label: {"Pretrained", "GeneClassifier", "CellClassifier"}
Thank you for your questions! We generally extract embeddings from the last layer of the model prior to the heads, which should encode aspects of all the tasks represented in the MTL training. The options for the model_type relate to how the model is loaded. For example, if you use the pretrained model directly, it's best to select "Pretrained". Otherwise, a randomly initialized head will be added to the top of the model if loaded as "CellClassifier" let's say, so embeddings extracted from the head would be random.