LaViDa-Pathgen

World's First Diffusion Model based Visual Language Model for Pathology based on LaViDa, trained on PathGen-1.6M dataset and finetuned on PathGen-Instruct datasets.

Github

Inference

Download checkpoint from https://huggingface.co/himanshunitrr/LaViDa-Pathgen You can infer using predict.py

Evaluation

PathMMU

To evaluate the model on PathMMU use main.py

Use the conda environment you created earlier for LLaVA for evaluating LLaVA based models and use the conda environment you created for LaViDa for evaluating LaViDa based models.

Also, for some reason for LLaVA based models, you need to use an old version of LLaVA, for more information, check this issue

• in the PathGen-LLaVA paper the reported accuracy is quite low (~60.1) but I got different results.

Thanks

A huge shoutout to @jacklishufan et al for LaViDa and answering all my stupid questions, @superjamessyx et al for PathGen and PathMMU and my Boss Anant for all the support and guidance.