|
|
--- |
|
|
library_name: transformers |
|
|
license: cc-by-nc-3.0 |
|
|
--- |
|
|
|
|
|
# Model Card for ReVision-250M-256-16-baseline |
|
|
|
|
|
This repository contains **ReVision-250M-256-16-baseline**, a compact **vision-language model (VLM)** designed for **Visual Instruction Rewriting**. The model rewrites **multimodal task-oriented instructions** into text-only commands, enabling privacy-preserving on-device AI by eliminating the need to process images in the cloud. |
|
|
|
|
|
## Key Features |
|
|
- **Lightweight (250M parameters)**: Designed for on-device deployment with efficient inference. |
|
|
- **Privacy-Preserving**: Converts multimodal inputs into structured text, reducing reliance on cloud-based processing. |
|
|
- **Fine-Tuned for Instruction Rewriting**: Trained on a dataset of 39,000 examples spanning 14 task-oriented domains. |
|
|
- **Compact Yet Effective**: Outperforms larger models like PaliGemma-v2 (10B) and QwenVL-7B in instruction rewriting tasks. |
|
|
|
|
|
## Model Architecture |
|
|
- **Vision Encoder**: `google/siglip-base-patch16-256` (processes 256×256 images). |
|
|
- **Language Model**: `OuteAI/Lite-Mistral-150M-v2-Instruct` (instruction-tuned). |
|
|
- **Multimodal Fusion**: Uses a linear projector to align vision and language embeddings. |
|
|
- **Training Dataset**: Pretrained on image captioning datasets (e.g., LLaVA-CC3M, LLaVA-Pretrain) and fine-tuned on the Visual Instruction Rewriting dataset. |
|
|
|
|
|
## Performance |
|
|
| Model | ROUGE-1 | BLEU | Intent Accuracy | Argument Similarity | |
|
|
|------------------------------------|---------|------|----------------|----------------------| |
|
|
| ReVision-250M-256-16-baseline | 56.9% | 27.7% | 56.5% | 68.8% | |
|
|
|
|
|
|
|
|
## How to Use |
|
|
|
|
|
### Install Dependencies |
|
|
```bash |
|
|
pip install torch transformers torchvision |
|
|
``` |
|
|
|
|
|
### Load the Model |
|
|
```bash |
|
|
from transformers import AutoProcessor, AutoModelForSeq2SeqLM |
|
|
import torch |
|
|
from PIL import Image |
|
|
|
|
|
# Load model and processor |
|
|
model_name = "hsiangfu/ReVision-250M-256-16-baseline" |
|
|
processor = AutoProcessor.from_pretrained(model_name) |
|
|
model = AutoModelForSeq2SeqLM.from_pretrained(model_name) |
|
|
|
|
|
# Prepare inputs (image + instruction) |
|
|
image = Image.open("example.jpg") |
|
|
instruction = "Call this number." |
|
|
|
|
|
inputs = processor(images=image, text=instruction, return_tensors="pt") |
|
|
outputs = model.generate(**inputs) |
|
|
|
|
|
# Decode rewritten instruction |
|
|
rewritten_instruction = processor.batch_decode(outputs, skip_special_tokens=True)[0] |
|
|
print("Rewritten Instruction:", rewritten_instruction) |
|
|
``` |
|
|
|
|
|
## Dataset |
|
|
|
|
|
The model was fine-tuned on the ReVision Multimodal Query Rewrites Dataset, a collection of 39,023 ⟨image, original instruction, rewritten instruction⟩ triplets covering: |
|
|
|
|
|
- Books: "Who wrote this book" → "Who wrote 'The Silent Patient'?" |
|
|
- Business Cards: "Call this number." → "Call 512-555-1234." |
|
|
- Flyers & Signboards: "Add this event to my calendar." → "Add 'Tech Conference' on May 5 at 2 PM to my calendar." |
|
|
- Landmarks: "Who made this?" → "Who made the Statue of Liberty?" |
|
|
- Products: "What brand is this product?" → "What brand made 'Mismatched Sandwich Cremes'?" |
|
|
- CD covers: "Who made this CD?" → "Who made 'Future'?" |
|
|
- Paintings: "Who is this painting by?" → "Who made the painting 'Mona Lisa'?" |
|
|
|
|
|
Link: https://huggingface.co/datasets/hsiangfu/multimodal_query_rewrites |
|
|
|
|
|
|
|
|
## Applications |
|
|
|
|
|
- AR/VR Assistants (e.g., Apple Vision Pro, Meta Ray-Ban Glasses) |
|
|
- Smartphones & Wearables (on-device AI assistants) |
|
|
- Accessibility & Assistive AI (for users with visual impairments) |
|
|
|
|
|
## Citation |
|
|
|
|
|
|
|
|
|
|
|
## Acknowledgments |
|
|
|
|
|
Developed by researchers at UT Austin and Yale University. Model and dataset are available for academic use. |
|
|
|
