|
|
--- |
|
|
license: apache-2.0 |
|
|
license_link: https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct/blob/main/LICENSE |
|
|
language: |
|
|
- ja |
|
|
- en |
|
|
tags: |
|
|
- vila |
|
|
- nvila |
|
|
- conversational |
|
|
- multimodal |
|
|
base_model: |
|
|
- Qwen/Qwen2.5-1.5B-Instruct |
|
|
- Efficient-Large-Model/paligemma-siglip-so400m-patch14-448 |
|
|
--- |
|
|
# Heron NVILA-Lite 2B |
|
|
|
|
|
Heron NVILA-Lite 2B is a vision language model trained for Japanese, based on the [NVILA](https://arxiv.org/abs/2412.04468)-Lite architecture. |
|
|
|
|
|
## Model Overview |
|
|
|
|
|
* **Developed by**: [Turing Inc.](https://www.turing-motors.com/) |
|
|
* **Vision Encoder**: [paligemma-siglip-so400m-patch14-448](https://huggingface.co/Efficient-Large-Model/paligemma-siglip-so400m-patch14-448) |
|
|
* **Projector**: mlp_downsample_2x2_fix |
|
|
* **Language Model**: [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) |
|
|
* **Language(s)**: Japanese, English |
|
|
|
|
|
## Setup |
|
|
|
|
|
```bash |
|
|
# I have confirmed that 4.46.0 and 4.49.0 also work. Other versions of Transformer may also work, but I have not tested them. |
|
|
pip install transformers==4.45.0 accelerate opencv-python torchvision einops pillow |
|
|
pip install git+https://github.com/bfshi/scaling_on_scales.git |
|
|
``` |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoConfig, AutoModel |
|
|
|
|
|
model_path = "turing-motors/Heron-NVILA-Lite-2B" |
|
|
|
|
|
# you can use config |
|
|
config = AutoConfig.from_pretrained(model_path, trust_remote_code=True) |
|
|
model = AutoModel.from_config(config, trust_remote_code=True, device_map="auto") |
|
|
|
|
|
# or directly from_pretrained |
|
|
model = AutoModel.from_pretrained(model_path, trust_remote_code=True, device_map="auto") |
|
|
|
|
|
# show chat_template |
|
|
print(model.tokenizer.chat_template) |
|
|
|
|
|
# examples generate with raw text |
|
|
response = model.generate_content(["こんにちは"]) |
|
|
print(response) |
|
|
print("---" * 40) |
|
|
|
|
|
# examples generate with text + image |
|
|
from PIL import Image |
|
|
import requests |
|
|
url = "http://images.cocodataset.org/val2017/000000039769.jpg" |
|
|
image = Image.open(requests.get(url, stream=True).raw).convert("RGB") |
|
|
response = model.generate_content([image, "画像を説明してください。"]) |
|
|
print(response) |
|
|
print("---" * 40) |
|
|
|
|
|
# examples generate using generation_config |
|
|
from PIL import Image |
|
|
import requests |
|
|
from transformers import GenerationConfig |
|
|
generation_config = { |
|
|
"max_new_tokens": 512, |
|
|
"temperature": 0.5, |
|
|
"do_sample": True, |
|
|
} |
|
|
generation_config = GenerationConfig(**generation_config) |
|
|
url = "http://images.cocodataset.org/val2017/000000039769.jpg" |
|
|
image = Image.open(requests.get(url, stream=True).raw).convert("RGB") |
|
|
response = model.generate_content( |
|
|
[image, "画像を説明してください。"], |
|
|
generation_config=generation_config |
|
|
) |
|
|
print(response) |
|
|
print("---" * 40) |
|
|
|
|
|
# examples generate with text + image + text + image + text |
|
|
from PIL import Image |
|
|
import requests |
|
|
url_list = [ |
|
|
"https://images.unsplash.com/photo-1694831404826-3400c48c188d?q=80&w=2070&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D", |
|
|
"https://images.unsplash.com/photo-1693240876439-473af88b4ed7?q=80&w=1974&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D" |
|
|
] |
|
|
images = [ |
|
|
Image.open(requests.get(url, stream=True).raw).convert("RGB") for url in url_list |
|
|
] |
|
|
response = model.generate_content([ |
|
|
images[0], |
|
|
"これは日本の横断歩道の画像です", |
|
|
images[1], |
|
|
"これはオーストリアの信号機の画像です", |
|
|
"各画像に写っている歩行者用信号機の色は何色ですか？"]) |
|
|
print(response) |
|
|
print("---" * 40) |
|
|
``` |
|
|
|
|
|
## Training Summary |
|
|
|
|
|
| Stage | Training | Data Sources | Samples | |
|
|
|--------|-------------------------------|-------------------------------|-------------| |
|
|
| Stage1 | Projector | [Japanese image text pairs](https://gitlab.llm-jp.nii.ac.jp/datasets/llm-jp-japanese-image-text-pairs), [LLaVA-Pretrain](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain) | 1.1M | |
|
|
| Stage2 | Projector, LLM | Filtered MOMIJI 3 snapshots (CC-MAIN-2024-46, CC-MAIN-2024-51, CC-MAIN-2025-05) | 13M | |
|
|
| | | [Japanese image text pairs (subset)](https://gitlab.llm-jp.nii.ac.jp/datasets/llm-jp-japanese-image-text-pairs), [Japanese interleaved data (subset)](https://gitlab.llm-jp.nii.ac.jp/datasets/llm-jp-japanese-interleaved-data), [mmc4-core (subset)](https://github.com/allenai/mmc4), [coyo-700m (subset)](https://huggingface.co/datasets/kakaobrain/coyo-700m), [wikipedia_ja](https://huggingface.co/datasets/turing-motors/Wikipedia-Vision-JA), [llava_pretrain_ja](https://huggingface.co/datasets/turing-motors/LLaVA-Pretrain-JA), [stair_captions](http://captions.stair.center/) | 20M | |
|
|
| Stage3 | Vision Encoder, Projector, LLM | [llava-instruct-v1_5-en-subset-358k](https://huggingface.co/datasets/llm-jp/llava-instruct-v1_5-en-subset-358k), [llava-instruct-ja](https://huggingface.co/datasets/llm-jp/llava-instruct-ja), [japanese-photos-conv](https://huggingface.co/datasets/llm-jp/japanese-photos-conversation), [ja-vg-vqa](https://huggingface.co/datasets/llm-jp/ja-vg-vqa-conversation), [synthdog-ja (subset)](https://huggingface.co/datasets/naver-clova-ix/synthdog-ja), [ai2d](https://huggingface.co/datasets/lmms-lab/ai2d), [synthdog-en](https://huggingface.co/datasets/naver-clova-ix/synthdog-en), [sherlock](https://github.com/allenai/sherlock) | 1.4M | |
|
|
|
|
|
## Evaluation |
|
|
I used [llm-jp-eval-mm](https://github.com/llm-jp/llm-jp-eval-mm) for this evaluation. All scores other than our models are taken from [llm-jp-eval-mm leaderboard](https://llm-jp.github.io/llm-jp-eval-mm/) and the [Asagi website](https://uehara-mech.github.io/asagi-vlm?v=1). |
|
|
|
|
|
| Model | LLM Size | Heron-Bench overall LLM (%) | JA-VLM-Bench-In-the-Wild LLM (/5.0) | JA-VG-VQA-500 LLM (/5.0) | |
|
|
|--------------------------------|---------|-----------------------------|-------------------------------------|--------------------------| |
|
|
| **Heron NVILA-Lite 2B** | 1.5B | 52.8 | 3.52 | 3.50 | |
|
|
| **Heron NVILA-Lite 15B** | 14B | 59.6 | 4.2 | 3.82 | |
|
|
| [LLaVA-CALM2-SigLIP](https://huggingface.co/cyberagent/llava-calm2-siglip) | 7B | 43.3 | 3.15 | 3.21 | |
|
|
| [Llama-3-EvoVLM-JP-v2](https://huggingface.co/SakanaAI/Llama-3-EvoVLM-JP-v2) | 8B | 39.3 | 2.92 | 2.96 | |
|
|
| [VILA-jp](https://huggingface.co/llm-jp/llm-jp-3-vila-14b) | 13B | 57.2 | 3.69 | 3.62 | |
|
|
| [Asagi-14B](https://huggingface.co/MIL-UT/Asagi-14B) | 13B | 55.8 | 3.44 | 3.84 | |
|
|
| [Qwen2-VL 7B Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) | 7B | 55.5 | 3.61 | 3.6 | |
|
|
| GPT-4o | - | 87.6 | 3.85 | 3.58 | |
|
|
|
|
|
## Risks and Limitations |
|
|
|
|
|
This model is experimental and has not been thoroughly calibrated for ethical compliance or legal standards. Caution is advised for sensitive applications. |
|
|
|
|
|
## License |
|
|
|
|
|
- Model weights are licensed under [Apache License 2.0](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct/blob/main/LICENSE). |
|
|
- Users must comply with [OpenAI terms of use](https://openai.com/policies/terms-of-use) due to the inclusion of GPT-4-generated synthetic data. |
|
|
|
|
|
## How to cite |
|
|
|
|
|
```bibtex |
|
|
@misc{HeronNVILALite2B, |
|
|
title = {Heron NVILA-Lite 2B}, |
|
|
author = {Shingo Yokoi}, |
|
|
year = {2025}, |
|
|
url = {https://huggingface.co/turing-motors/Heron-NVILA-Lite-2B}, |
|
|
} |
|
|
``` |
|
|
|
|
|
## Citations |
|
|
|
|
|
```bibtex |
|
|
@misc{liu2025nvilaefficientfrontiervisual, |
|
|
title={NVILA: Efficient Frontier Visual Language Models}, |
|
|
author={Zhijian Liu and Ligeng Zhu and Baifeng Shi and Zhuoyang Zhang and Yuming Lou and Shang Yang and Haocheng Xi and Shiyi Cao and Yuxian Gu and Dacheng Li and Xiuyu Li and Yunhao Fang and Yukang Chen and Cheng-Yu Hsieh and De-An Huang and An-Chieh Cheng and Vishwesh Nath and Jinyi Hu and Sifei Liu and Ranjay Krishna and Daguang Xu and Xiaolong Wang and Pavlo Molchanov and Jan Kautz and Hongxu Yin and Song Han and Yao Lu}, |
|
|
year={2025}, |
|
|
eprint={2412.04468}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.CV}, |
|
|
url={https://arxiv.org/abs/2412.04468}, |
|
|
} |
|
|
``` |
