metadata

datasets:
  - trinhvg/ViDRiP_Instruct_Test
  - trinhvg/ViDRiP_Instruct_Train

🧬 ViDRiP-LLaVA: A Dataset and Benchmark for Diagnostic Reasoning from Pathology Videos

ViDRiP-LLaVA is a vision-language framework designed for instruction-based diagnostic reasoning using both image patches and video clips from pathology slides. It builds on LLaVA and extends it to the medical domain with domain-specific datasets and fine-tuned models.

🧠 Introducing our ViDRiP-LLaVA: the first multimodal model for diagnostic reasoning in pathology through video-based instruction. 🔬📽️

Our method leverages chain-of-thought (CoT) prompting to distill the reasoning capabilities of LLMs. ViDRiP-LLaVA generates both detailed histological descriptions and final diagnoses, simulating how pathologists analyze and sign out cases.

📚 Trained on 4,278 instructional video pairs

⚙️ Combines single-image + clip transfer and fine-tuning on segmented diagnostic videos

📚 Video Datasets

🔹 ViDRiP_Instruct_Train

The videos data is ~ 100 GB:

🔹 ViDRiP_Instruct_Train_Video_Hugging Face (There is 10 zip files)

4,000+ instruction-style samples
Each sample includes:
- A pathology video clip
- A diagnostic question
- A multi-turn reasoning answer
Format: JSON + MP4
Croissant-compliant metadata for structured use

🔹 ViDRiP_Instruct_Test

🔹 ViDRiP_Instruct_Test_Video

Held-out test set of diagnostic Q&A pairs
Used for benchmarking reasoning performance

📚 Image Datasets

We use publicly available datasets: Quilt-LLaVA and PathAsst. Please refer to their respective repositories for download instructions.

Quilt-LLaVA: A vision-language dataset for pathology adapted from LLaVA.
PathAsst: A generative assistant for pathology with curated image-text pairs.

🤖 Models

🔸 ViDRiP_LLaVA_video

Vision-language model for video-based diagnostic reasoning
Trained on ViDRiP_Instruct_Train
Suitable for:
- Medical VQA
- Instructional explanation generation
- Educational pathology summarization

🔸 ViDRiP_LLaVA_image

Vision-language model for patch-based diagnostic prompts
Useful for pathology captioning and single-frame inference

🚀 Quickstart

🔧 Fine-tuning the model on video dataset

./scripts/train/finetune_ov_video.sh

🪄 Fine-tuning with LoRA

./scripts/train/finetune_ov_video_lora.sh

🔗 Merge LoRA weights

./scripts/train/merge_lora_weights.py

🧪 Usage / Demo

./doc/ViDRiP_LLaVA_trial.py

🔧 Evaluate on our video dataset

We use lmms_eval to evaluate the performance of video diagnostic reasoning.

To benchmark ViDRiP-LLaVA and compare it with other models:

Clone the lmms_eval repo
Copy our evaluation task folder into it:

cp -r lmms_eval/tasks/ViDRiP_Instruct_Test /path/to/lmms_eval/tasks/

You can then run evaluation using the standard lmms_eval CLI interface.

Citation:

Coming soon

📄 Usage and License Notices

ViDRiP-LLaVA (Vision-language Diagnostic Reasoning in Pathology), including its dataset, code, and model checkpoints, is released strictly for non-commercial research purposes only.

📁 Licenses

Dataset: Licensed under CC BY-NC-ND 3.0 (Attribution–NonCommercial–NoDerivatives)
Code and pretrained models: Licensed under CC BY-NC 3.0 (Attribution–NonCommercial)

⚙️ Dependencies and Components

This project may incorporate or build upon resources such as LLaVA-Next, QUILT-1M, LLaMA, PathAsst, and GPT-4, each subject to their own licenses and Terms of Use.

🎥 Source Acknowledgment

ViDRiP-LLaVA includes data derived from public educational pathology videos hosted on YouTube. All content usage complies with YouTube’s Terms of Service, and the intellectual property rights of the original pathologist creators are fully acknowledged and respected.

🚫 Restrictions

Not for commercial use
Not to be used in clinical care or medical decision-making
For academic research, development, and evaluation only