---
base_model: unsloth/llama-3.2-11b-vision-instruct-unsloth-bnb-4bit
tags:
- text-generation-inference
- transformers
- unsloth
- mllama
license: apache-2.0
language:
- en
---

# Fine-tuned Vision-Language Model for Radiology Report Generation

This repository contains a fine-tuned vision-language model for generating radiology reports. It's based on the [Unsloth](https://github.com/unslothai/unsloth) library and utilizes the Llama-3.2-11B-Vision-Instruct model as a base. 

## Model Description

This model is fine-tuned on a sampled version of the ROCO radiography dataset ([Radiology_mini](https://huggingface.co/datasets/unsloth/Radiology_mini)). It's designed to assist medical professionals by providing accurate descriptions of medical images, such as X-rays, CT scans, and ultrasounds. 

The fine-tuning process uses Low-Rank Adaptation (LoRA) to efficiently train the model, focusing on the language layers while keeping the vision layers frozen. This approach minimizes the computational resources required for fine-tuning while achieving significant performance improvements.

## Usage

To use this model, you'll need the Unsloth library:

```bash
pip install unsloth
```

Then, you can load the model and tokenizer:

```python
from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained("awaliuddin/unsloth_finetune", load_in_4bit=True)
FastVisionModel.for_inference(model)
```

```python
from PIL import Image

image = Image.open("path/to/your/image.jpg") # Replace with your image path
instruction = "You are an expert radiographer. Describe accurately what you see in this image."
messages = [ {"role": "user", "content": [ {"type": "image"}, {"type": "text", "text": instruction} ]} ]

input_text = tokenizer.apply_chat_template(messages, add_generation_prompt=True) inputs = tokenizer(image, input_text, add_special_tokens=False, return_tensors="pt").to("cuda")

from transformers import TextStreamer

text_streamer = TextStreamer(tokenizer, skip_prompt=True) _ = model.generate(**inputs, streamer=text_streamer, max_new_tokens=128, use_cache=True, temperature=1.5, min_p=0.1)
```

## Training Details

* **Base Model:** Llama-3.2-11B-Vision-Instruct
* **Dataset:** Radiology_mini (sampled from ROCO radiography dataset)
* **Fine-tuning Method:** LoRA (language layers only)
* **Optimizer:** AdamW 8-bit
* **Learning Rate:** 2e-4

## Limitations

* This model is trained on a limited dataset and might not generalize well to all types of medical images.
* The generated reports should be reviewed by qualified medical professionals before being used for diagnostic purposes.

## Acknowledgements

* The Unsloth library for efficient fine-tuning of vision-language models.
* The Hugging Face team for providing the platform and tools for model sharing.
* The authors of the ROCO radiography dataset.

## License

[Apache-2.0 License]

# Uploaded finetuned  model

- **Developed by:** Awaliuddin
- **License:** apache-2.0
- **Finetuned from model :** unsloth/llama-3.2-11b-vision-instruct-unsloth-bnb-4bit

This mllama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.

[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)