---
license: apache-2.0
tags:
  - medical
  - summarization
  - lora
  - mistral
  - dialogue
  - peft
model-index:
  - name: Fine-tuned Mistral-7B (LoRA) on MTS-Dialog
    results:
      - task:
          type: summarization
        metrics:
          - name: ROUGE-1
            type: rouge
            value: 0.1318
          - name: ROUGE-2
            type: rouge
            value: 0.0456
          - name: ROUGE-L
            type: rouge
            value: 0.0900
          - name: BLEU
            type: bleu
            value: 0.0260
---

# 🧠 LoRA Fine-Tuned Mistral-7B on MTS-Dialog

This repository contains a LoRA fine-tuned version of [`mistralai/Mistral-7B-v0.1`](https://huggingface.co/mistralai/Mistral-7B-v0.1) for medical dialogue summarization, trained on the [MTS-Dialog](https://github.com/abachaa/MTS-Dialog) dataset.

---

## 🔗 Resources

- 📘 Training Notebook & Code: [GitHub Repository](https://github.com/imsachinsingh00/LoRA_Mistral_MTSDialog_Summarization)

---

## 📘 Model Summary

- **Base Model**: `mistralai/Mistral-7B-v0.1`
- **Fine-tuning Method**: LoRA (Low-Rank Adaptation)
- **Frameworks**: 🤗 Transformers, PEFT, bitsandbytes
- **Quantization**: 4-bit
- **Task**: Medical dialogue summarization
- **Dataset**: [MTS-Dialog](https://github.com/abachaa/MTS-Dialog)

---

## 🏥 Task Description

This model is trained to summarize doctor-patient conversations into concise clinical notes, categorized by sections such as `GENHX`, `HPI`, `ROS`, etc. These summaries assist with EHR documentation and clinical decision-making.

---

## ⚙️ Training Configuration

| Parameter          | Value         |
|--------------------|---------------|
| LoRA Rank          | 4             |
| Epochs             | 3             |
| Batch Size         | 4 (×4 grad. acc.) |
| Learning Rate      | 3e-4          |
| Device             | CUDA:0        |
| Quantization       | 4-bit (bnb)   |

> ⚠️ Due to limited GPU resources (office laptop), training was constrained to 3 epochs and a small LoRA rank. Performance is expected to improve significantly with extended training and better hardware.

---

## 📊 Evaluation Metrics

| Metric    | Score  |
|-----------|--------|
| ROUGE-1   | 0.1318 |
| ROUGE-2   | 0.0456 |
| ROUGE-L   | 0.0900 |
| BLEU      | 0.0260 |

---

## 💡 Example Prompt

```text
Summarize the following dialogue for section: GENHX
Doctor: What brings you back into the clinic today, miss?
Patient: I've had chest pain for the last few days.
Doctor: When did it start?
Summary:


## 🧪 Inference Code

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", load_in_4bit=True)
model = PeftModel.from_pretrained(model, "Imsachinsingh00/Fine_tuned_LoRA_Mistral_MTSDialog_Summarization")
model.eval()

tokenizer = AutoTokenizer.from_pretrained("Imsachinsingh00/Fine_tuned_LoRA_Mistral_MTSDialog_Summarization")

prompt = "Summarize the following dialogue for section: HPI\nDoctor: Hello, what brings you in?\nPatient: I've been dizzy for two days.\nSummary:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=150)
print(tokenizer.decode(output[0], skip_special_tokens=True))


## 📁 Included Files

- `config.json` – PEFT configuration for LoRA  
- `adapter_model.bin` – LoRA adapter weights  
- `tokenizer/` – Tokenizer files  
- `README.md` – This model card  

## 📌 Notes

- 🚫 This is not a fully optimized clinical model — only a proof of concept.  
- 💡 Consider training longer (`epochs=10`, `rank=8`) on GPUs with higher VRAM for better results.