Upload README.md with huggingface_hub

README.md

@@ -1 +1,98 @@
 
+# 🧬 ViDRiP-LLaVA: Multimodal Diagnostic Reasoning in Pathology
+
+**ViDRiP-LLaVA** is a vision-language framework designed for instruction-based diagnostic reasoning using both image patches and video clips from pathology slides. It builds on LLaVA and extends it to the medical domain with domain-specific datasets and fine-tuned models.
+
+
+🧠 Introducing our ViDRiP-LLaVA: the first multimodal model for diagnostic reasoning in pathology through video-based instruction. 🔬📽️
+
+Our method leverages chain-of-thought (CoT) prompting to distill the reasoning capabilities of LLMs. ViDRiP-LLaVA generates both detailed histological descriptions and final diagnoses, simulating how pathologists analyze and sign out cases.
+
+📚 Trained on 4,278 instructional video pairs
+
+⚙️ Combines single-image + clip transfer and fine-tuning on segmented diagnostic videos
+
+
+---
+<p align="center" width="100%">
+<img src="assets/Network.png"  width="80%" height="80%">
+</p>
+
+
+## 📚 Datasets
+
+### 🔹 [ViDRiP_Instruct_Train](https://huggingface.co/datasets/trinhvg/ViDRiP_Instruct_Train)
+### 🔹 [ViDRiP_Instruct_Train_Video](https://drive.google.com/drive/folders/1oxZlaJpE7PGDYt32LeoGgIzwEvWdnupY?usp=sharing)
+- 4,000+ instruction-style samples
+- Each sample includes:
+  - A pathology video clip
+  - A diagnostic question
+  - A multi-turn reasoning answer
+- Format: JSON + MP4
+- Croissant-compliant metadata for structured use
+
+### 🔹 [ViDRiP_Instruct_Test](https://huggingface.co/datasets/trinhvg/ViDRiP_Instruct_Test)
+### 🔹 [ViDRiP_Instruct_Test_Video](https://drive.google.com/drive/folders/1oxZlaJpE7PGDYt32LeoGgIzwEvWdnupY?usp=sharing)
+
+- Held-out test set of diagnostic Q&A pairs
+- Used for benchmarking reasoning performance
+
+---
+
+## 🤖 Models
+
+### 🔸 [ViDRiP_LLaVA_video](https://huggingface.co/trinhvg/ViDRiP_LLaVA_video)
+
+- Vision-language model for video-based diagnostic reasoning
+- Trained on `ViDRiP_Instruct_Train`
+- Suitable for:
+  - Medical VQA
+  - Instructional explanation generation
+  - Educational pathology summarization
+
+### 🔸 [ViDRiP_LLaVA_image](https://huggingface.co/trinhvg/ViDRiP_LLaVA_image)
+
+- Vision-language model for patch-based diagnostic prompts
+- Useful for pathology captioning and single-frame inference
+
+
+
+
+## 🚀 Quickstart
+
+### 🔧 Fine-tuning the model on video dataset
+```bash
+./scripts/train/finetune_ov_video.sh
+```
+
+### 🪄 Fine-tuning with LoRA
+```bash
+./scripts/train/finetune_ov_video_lora.sh
+```
+🔗 Merge LoRA weights
+```bash
+./scripts/train/merge_lora_weights.py
+```
+### 🧪 Usage / Demo
+```bash
+./doc/ViDRiP_LLaVA_trial.py
+```
+
+
+### 🔧 Evaluate on our video dataset
+
+We use [lmms_eval](https://github.com/EvolvingLMMs-Lab/lmms-eval) to evaluate the performance of video diagnostic reasoning.
+
+To benchmark `ViDRiP-LLaVA` and compare it with other models:
+
+1. Clone the `lmms_eval` repo
+2. Copy our evaluation task folder into it:
+
+```bash
+cp -r lmms_eval/tasks/ViDRiP_Instruct_Test /path/to/lmms_eval/tasks/
+```
+You can then run evaluation using the standard lmms_eval CLI interface.
+
+
+### Citation:
+Coming soon