---
license: other
license_name: license.md
license_link: LICENSE
base_model:
- nvidia/segformer-b3-finetuned-cityscapes-1024-1024
tags:
- cancer_research,
- biology,
- microscopy
---

See github here for usage:
https://github.com/ReyaLab/AI_Organoid_Counter

https://reya-lab.org/

We have since updated to [this model](https://huggingface.co/ReyaLabColumbia/Segformer_Organoid_Counter_GP).

Custom fine-tuned version of NVIDIA's segformer model for colony slides in microscopy.

Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021). SegFormer: Simple and efficient design for semantic segmentation with transformers. arXiv preprint arXiv:2105.15203. https://arxiv.org/abs/2105.15203

This model takes 512x512 images for segmentation. 

The Segformer-Cityscapes model was changed to a ternary classifier and fine-tuned on custom training data, where organoids and necrosis were made as separate masks and then merged with different grayscale values.
![test12.png](https://cdn-uploads.huggingface.co/production/uploads/6813e81cfd7a2af93d0e5384/T1cgAw1cnbzXOQbujcPfj.png)
![Mask_test12.png](https://cdn-uploads.huggingface.co/production/uploads/6813e81cfd7a2af93d0e5384/Bu-JLpMzYv1UqZVr5BzUz.png)
![Mask2_test12.png](https://cdn-uploads.huggingface.co/production/uploads/6813e81cfd7a2af93d0e5384/etghwv4AEr15uceSk1upQ.png)

After training, it was able to correctly identify organoids and necrosis.
![example2.png](https://cdn-uploads.huggingface.co/production/uploads/6813e81cfd7a2af93d0e5384/-Igd3jirHou-ChM3ZHZvo.png)
![example.png](https://cdn-uploads.huggingface.co/production/uploads/6813e81cfd7a2af93d0e5384/N3b_rdSfWec4ybIB6Ijfq.png)

The python program (see linked GitHub) then uses the masks to annotate the images and provide statistics about the colonies.
![Group_analysis_results3.png](https://cdn-uploads.huggingface.co/production/uploads/6813e81cfd7a2af93d0e5384/dUcZcjmMp4jbRsSgQJLiD.png)