metadata
license: apache-2.0
datasets:
- AIDC-AI/Ovis-dataset
library_name: transformers
tags:
- MLLM
pipeline_tag: image-text-to-text
Introduction
Ovis is a novel Multimodal Large Language Model (MLLM) architecture, designed to structurally align visual and textual embeddings. For a comprehensive introduction, please refer to Ovis paper and Ovis GitHub.
Model
Ovis can be instantiated with popular LLMs (e.g., Qwen, Llama3). We provide the following pretrained Ovis MLLMs:
| Ovis-Clip-Qwen1.5-7B | Ovis-Clip-Llama3-8B | Ovis-Clip-Qwen1.5-14B | |
|---|---|---|---|
| ViT | Clip | Clip | Clip |
| LLM | Qwen1.5-7B-Chat | Llama3-8B-Instruct | Qwen1.5-14B-Chat |
| Download | Huggingface | Huggingface | Huggingface |
| MMStar | 44.3 | 49.5 | 48.5 |
| MMB-EN | 75.1 | 77.4 | 78.4 |
| MMB-CN | 70.2 | 72.8 | 76.6 |
| MMMU-Val | 39.7 | 44.7 | 46.7 |
| MMMU-Test | 37.7 | 39.0 | 40.7 |
| MathVista-Mini | 41.4 | 40.8 | 43.4 |
| MME | 1882 | 2009 | 1961 |
| HallusionBench | 56.4 | 61.1 | 57.6 |
| RealWorldQA | 60.0 | 57.9 | 62.7 |
Usage
Below is a code snippet to run Ovis with multimodal inputs. For additional usage instructions, including inference wrapper and Gradio UI, please refer to Ovis GitHub.
pip install torch==2.1.0 transformers==4.41.1 deepspeed==0.14.0 pillow==10.3.0
import torch
from PIL import Image
from transformers import AutoModelForCausalLM
# load model
model = AutoModelForCausalLM.from_pretrained("AIDC-AI/Ovis-Clip-Llama3-8B",
torch_dtype=torch.bfloat16,
multimodal_max_length=8192,
trust_remote_code=True).cuda()
text_tokenizer = model.get_text_tokenizer()
visual_tokenizer = model.get_visual_tokenizer()
conversation_formatter = model.get_conversation_formatter()
# enter image path and prompt
image_path = input("Enter image path: ")
image = Image.open(image_path)
text = input("Enter prompt: ")
query = f'<image> {text}'
prompt, input_ids = conversation_formatter.format_query(query)
input_ids = torch.unsqueeze(input_ids, dim=0).to(device=model.device)
attention_mask = torch.ne(input_ids, text_tokenizer.pad_token_id).to(device=model.device)
pixel_values = [visual_tokenizer.preprocess_image(image).to(
dtype=visual_tokenizer.dtype, device=visual_tokenizer.device)]
# print model output
with torch.inference_mode():
kwargs = dict(
pixel_values=pixel_values,
attention_mask=attention_mask,
do_sample=False,
top_p=None,
temperature=None,
top_k=None,
repetition_penalty=None,
max_new_tokens=512,
use_cache=True,
eos_token_id=text_tokenizer.eos_token_id,
pad_token_id=text_tokenizer.pad_token_id
)
output_ids = model.generate(input_ids, **kwargs)[0]
input_token_len = input_ids.shape[1]
output = text_tokenizer.decode(output_ids[input_token_len:], skip_special_tokens=True)
print(f'Output: {output}')
Citation
If you find Ovis useful, please cite the paper
@article{lu2024ovis,
title={Ovis: Structural Embedding Alignment for Multimodal Large Language Model},
author={Shiyin Lu and Yang Li and Qing-Guo Chen and Zhao Xu and Weihua Luo and Kaifu Zhang and Han-Jia Ye},
year={2024},
journal={arXiv:2405.20797}
}
License
The project is licensed under the Apache 2.0 License and is restricted to uses that comply with the license agreements of Qwen, Llama3, and Clip.