Update README.md

README.md

@@ -4,7 +4,7 @@ language:
 - zh
 - en
 ---
-# PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understanding
+# Pathology-LLaVA-(PCaption-0.5M dataset) 
 
 We developed a domain-speciffc large language-vision assistant (PA-LLaVA) for pathology image understanding. Specifically, (1) we first construct a human pathology image-text dataset by cleaning the public medical image-text data for domainspecific alignment; (2) Using the proposed image-text data, we first train a pathology language-image pretraining (PLIP) model as the specialized visual encoder for pathology image, and then we developed scale-invariant connector to avoid the information loss caused by image scaling; (3) We adopt two-stage learning to train PA-LLaVA, first stage for domain alignment, and second stage for end to end visual question & answering (VQA) task.
 
@@ -16,10 +16,18 @@ We developed a domain-speciffc large language-vision assistant (PA-LLaVA) for pa
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/663f06e01cd68975883a353e/SkFB0x3JunWE_Wae808Nq.png)
 
 
-## Data Cleaning Process
+## Human Pathology Image-Text data （PCaption-0.5M）
+
+### Introduction
+These public datasets contain substantial amounts of data unrelated to human pathology. To obtain the human pathology image-text data, we performed two cleaning processes on the raw data, as illustrated in the follow figture: (1) Removing nonpathological images. (2) Removing nonhuman pathology data. Additionally, we excluded image-text pairs with textual descriptions of fewer than 20 words. Ultimately, we obtained 518,413 image-text pairs (named "PCaption-0.5M" ) for the aligned training dataset.
+
+Instruction fine-tuning phase we only cleaned PMC-VQA in the same way and obtained 15,788 question-answer pairs related to human pathology. Lastly, we combined PathVQA and Human pathology data obtained from PMC-VQA, thereby constructing a dataset of 35543 question-answer pairs.data.
+
+#### Data Cleaning Process
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/663f06e01cd68975883a353e/IAeFWhH8brZYDaTJnew2N.png)
 
+### Get the Dataset
 
 ### Step 1 Download the public datasets.
 Here we only provide the download link for the public dataset and expose the image id index of our cleaned dataset on HuggingFace.
@@ -65,7 +73,7 @@ Finally, run dataformate.py to get the format needed to train the model.
 python dataformat.py
 ```
 
-## Model
+## Checkpoint
 Our released weights are distributed training weights that can be directly loaded for training through XTuner. If you need merged weights, they can be merged using XTuner (using the weights from the domain alignment phase as an example):
 ```
 xtuner convert pth_to_hf path/pallava_domain_alignment.py ./domain_alignment_weight.pth ./domain_alignment_weight_ft
@@ -92,6 +100,19 @@ NPROC_PER_NODE=8 NNODES=2 PORT=12345 ADDR= NODE_RANK=0 xtuner train pallava_inst
 
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/663f06e01cd68975883a353e/ng9KbUevJk5HyYONpOg2S.png)
 
+## Citation
+```
+@misc{dai2024pallavalargelanguagevisionassistant,
+      title={PA-LLaVA: A Large Language-Vision Assistant for Human Pathology Image Understanding}, 
+      author={Dawei Dai and Yuanhui Zhang and Long Xu and Qianlan Yang and Xiaojing Shen and Shuyin Xia and Guoyin Wang},
+      year={2024},
+      eprint={2408.09530},
+      archivePrefix={arXiv},
+      primaryClass={cs.AI},
+      url={https://arxiv.org/abs/2408.09530}, 
+}
+```
+
 ## contact
 
 mailto: [[email protected]](mailto:[email protected]) or [[email protected]](mailto:[email protected])