🧠 MedCLIP-Glioma-LoRA-8bit

A fine-tuned version of PubMedCLIP using LoRA adapters on the BRATS-derived Glioma dataset for highly precise brain tumor classification.

---

🔍 Model Overview

This model builds upon pre-trained PubMedCLIP — a CLIP variant pre-trained on radiology images from the ROCO dataset — and adapts it for Glioma tumor classification using supervised fine-tuning (SFT) with Low-Rank Adaptation (LoRA) and 8-bit quantization.

• Base model: flaviagiammarino/pubmed-clip-vit-base-patch32
• Fine-tuned on: BRATS-derived dataset from The Cancer Imaging Archive (TCIA)
• Preprocessing: Max Intensity Projection (MIP)
• Format: DICOM & NIfTI ➝ PNG (via 2D slicing)
• Adaptation: LoRA applied to attention and MLP layers
• Precision: Quantized from FP32 → 8-bit using bitsandbytes

---

📁 Dataset

• Name: parthsalke/vlm_glioma_dataset
• Modalities: T1, T1-CE, T2, and T2 FLAIR MRI
• Format: Preprocessed from DICOM/NIFTI to RGB PNG
• Pairs: Image + Text

---

🧪 Fine-Tuning Strategy

Component	Strategy
Vision Encoder	Frozen base, LoRA adapters (qkv)
Text Encoder	Frozen base, LoRA adapters
Training Method	Supervised Fine-Tuning (SFT)
Optimizer	AdamW
LoRA Params	r=8, alpha=32, dropout=0.1
Precision	8-bit (bnb + PEFT)

---

🧪 How to Use

from transformers import CLIPProcessor
from peft import PeftModel
from PIL import Image

# Load base + LoRA adapter
from transformers import CLIPModel
base = CLIPModel.from_pretrained("flaviagiammarino/pubmed-clip-vit-base-patch32")
model = PeftModel.from_pretrained(base, "parthsalke/biomedclip-glioma-lora-8bit")

# Load processor
processor = CLIPProcessor.from_pretrained("flaviagiammarino/pubmed-clip-vit-base-patch32")

# Inference example
image = Image.open("example.png").convert("RGB")
text = [" ", " "]  #Input labels here

inputs = processor(images=image, text=text, return_tensors="pt", padding=True)
outputs = model(**inputs)
probs = outputs.logits_per_image.softmax(dim=1)