Hugging Face's logo

BytedanceDouyinContent
/
SAIL-VL-1d5-2B

Safetensors
English
Chinese
Model card Files Community
2

Introduction

lidar_map

SAIL-VL is a state-of-the-art vision-language model (VLM) developed by the Bytedance Douyin Content Team. The goal of SAIL-VL is to develope a high-performance vision language model that facilitates deployment on mobile devices and ensures accessibility and affordability for a broad audience. Through careful tuning of data and training recipes, SAIL-VL demonstrates that even a small VLM can benefit significantly from data scaling.

SAIL-VL V1.5 is the brand new version of our model, which incorporates advanced techniques to achieve higher performance. For visual encoding, we use the stronger AIM-V2 ViT as our vision encoder, introduce the progressive training strategy to warmup and a visual token scaling strategy during inference. During training, we introduce an adaptive stream packing strategy to support higher throughput and longer sequences. Finally, we add more conversation and reasoning data, filter out noisy data and add a new training stage for videos. With all these updates, our model outperforms recent SoTA models of comparable sizes, InternVL-3-2B, Ovis2-2B and even Qwen2.5-VL-3B.

Please enjoy our model and feel free to contact us for any question or opportunity.

News🚀🚀🚀

Model Card

Model Architecture:

Architecture ViT LLM Adapter Token Merge Resolution
🤗SAIL-VL-1.5-2B 🤗AimV2-Huge 🤗Qwen2.5-1.5B 2-layer MLP 2x2 448x448xN
🤗SAIL-VL-1.5-8B 🤗InternViT-300M 🤗Qwen2.5-7B 2-layer MLP 2x2 448x448xN
🤗SAIL-VL-2B 🤗InternViT-300M 🤗Qwen2.5-1.5B 2-layer MLP 2x2 448x448xN
🤗SAIL-VL-8B 🤗InternViT-300M 🤗Qwen2.5-7B 2-layer MLP 2x2 448x448xN

Training Recipes Overview:

Sail-VL benefits from high-quality data and carefully curated training recipes. We find the data quality, quantity and the design of curriculum training pipeline are crucial for model performance. With the proper design and data, the model's capacity scales effectively with data expansion at all stages, leading to enhanced performance.

Evaluation

SAIL-VL is competitive compared with recently released SoTAs, InternVL-2.5, Qwen2.5-VL, Ovis-2 and InternVL3.

Detail Evaluations:

Benchmark InternVL-2.5-2B Qwen2.5-VL-3B InternVL3-2B Ovis2-2B SAIL-VL-1.5-2B
OpenCompassAvg 60.78 64.90 64.83 65.49 67.67
Total Avg 66.54 71.35 69.97 70.30 72.61
GeneralQA Avg 63.55 66.83 68.34 64.05 68.05
OCR Avg 76.79 83.25 79.27 83.09 83.74
MMBench_DEV_V11 * 79.73 82.40 84.25 81.79 85.05
MathVista_MINI 52.00 60.20 57.30 64.30 67.30
MMStar * 53.40 55.13 61.33 58.13 62.80
MMMU_VAL * 41.89 48.11 47.11 42.67 42.89
MMVet 61.33 61.24 65.14 57.39 61.38
HallusionBench 42.79 48.29 41.42 50.05 49.80
AI2D_TEST + 74.90 80.73 78.72 82.80 83.68
OCRBench + 802 830 834 868 885
RealWorldQA * 61.05 65.36 64.58 66.41 67.06
NaturalBench * 69.95 72.49 75.34 62.97 76.80
InfoVQA_VAL + 61.85 76.11 66.94 72.70 71.82
ChartQA_TEST + 79.36 87.00 80.40 84.72 84.84
MME * 75.25 77.46 77.44 72.32 73.67
DocVQA_VAL + 87.68 93.11 87.46 91.49 91.62
TextVQA_VAL + 76.76 79.55 78.71 80.00 81.98

Details for average performance section:

  • OpenCompass-Avg includes public avaliable validation sets from OpenCompass: AI2D_TEST, HallusionBench, MMBench_DEV_V11, MMMU_DEV_VAL, MMStar, MMVet, MathVista_MINI, evaluated by our team.

  • GeneralQA-Avg includes datasets with "*" mark.

  • OCR-Avg includes datasets with "+" mark.

How to Use

The basic usage and dynamic crop strategy of SAIL-VL follows InternVL2, you can easily switch Intern-VL series of models to our model. Here is a simple example of using our model:

Requirements: 

pip3 install einops transformers timm

Code: 

import numpy as np
import torch
import torchvision.transforms as T
from PIL import Image
from torchvision.transforms.functional import InterpolationMode
from transformers import AutoModel, AutoTokenizer

IMAGENET_MEAN = (0.485, 0.456, 0.406)
IMAGENET_STD = (0.229, 0.224, 0.225)

def build_transform(input_size):
    MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
    transform = T.Compose([
        T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
        T.ToTensor(),
        T.Normalize(mean=MEAN, std=STD)
    ])
    return transform

def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
    best_ratio_diff = float('inf')
    best_ratio = (1, 1)
    area = width * height
    for ratio in target_ratios:
        target_aspect_ratio = ratio[0] / ratio[1]
        ratio_diff = abs(aspect_ratio - target_aspect_ratio)
        if ratio_diff < best_ratio_diff:
            best_ratio_diff = ratio_diff
            best_ratio = ratio
        elif ratio_diff == best_ratio_diff:
            if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
                best_ratio = ratio
    return best_ratio

def dynamic_preprocess(image, min_num=1, max_num=10, image_size=448, use_thumbnail=False):
    orig_width, orig_height = image.size
    aspect_ratio = orig_width / orig_height

    # calculate the existing image aspect ratio
    target_ratios = set(
        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
        i * j <= max_num and i * j >= min_num)
    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])

    # find the closest aspect ratio to the target
    target_aspect_ratio = find_closest_aspect_ratio(
        aspect_ratio, target_ratios, orig_width, orig_height, image_size)

    # calculate the target width and height
    target_width = image_size * target_aspect_ratio[0]
    target_height = image_size * target_aspect_ratio[1]
    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]

    # resize the image
    resized_img = image.resize((target_width, target_height))
    processed_images = []
    for i in range(blocks):
        box = (
            (i % (target_width // image_size)) * image_size,
            (i // (target_width // image_size)) * image_size,
            ((i % (target_width // image_size)) + 1) * image_size,
            ((i // (target_width // image_size)) + 1) * image_size
        )
        # split the image
        split_img = resized_img.crop(box)
        processed_images.append(split_img)
    assert len(processed_images) == blocks
    if use_thumbnail and len(processed_images) != 1:
        thumbnail_img = image.resize((image_size, image_size))
        processed_images.append(thumbnail_img)
    return processed_images

def load_image(image_file, input_size=448, max_num=10):
    image = Image.open(image_file).convert('RGB')
    transform = build_transform(input_size=input_size)
    images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
    pixel_values = [transform(image) for image in images]
    pixel_values = torch.stack(pixel_values)
    return pixel_values

path = "BytedanceDouyinContent/SAIL-VL-1.5-2B"
model = AutoModel.from_pretrained(
    path,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True).eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)

# set the max number of tiles in `max_num`
pixel_values = load_image('./test.png', max_num=10).to(torch.bfloat16).cuda()
generation_config = dict(max_new_tokens=1024, do_sample=True)

# pure-text conversation
question = 'Hello, who are you?'
response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)
print(f'User: {question}         Assistant: {response}')

question = 'Can you tell me a story?'
response, history = model.chat(tokenizer, None, question, generation_config, history=history, return_history=True)
print(f'User: {question}         Assistant: {response}')

# single-image single-round conversation
question = '<image>         Please describe the image shortly.'
response = model.chat(tokenizer, pixel_values, question, generation_config)
print(f'User: {question}         Assistant: {response}')

# single-image multi-round conversation
question = '<image>         Please describe the image in detail.'
response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)
print(f'User: {question}         Assistant: {response}')

question = 'Please write a poem according to the image.'
response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True)
print(f'User: {question}         Assistant: {response}')

Acknowledge

Our model is built upon numerous outstanding open-source projects, and we are grateful for their contributions. We extend special thanks to the InternVL team and Qwen team for their great base models, and to the BAAI team (Infinity-MM) for their generous release of data.

Citation 

@article{dong2025scalable,
  title={Scalable vision language model training via high quality data curation},
  author={Dong, Hongyuan and Kang, Zijian and Yin, Weijie and Liang, Xiao and Feng, Chao and Ran, Jiao},
  journal={arXiv preprint arXiv:2501.05952},
  year={2025}
}

@misc{
    sailvl,
    title = {SAIL-VL: Scalable Vision Language Model Training with High Quality Data Curation},
    url = {https://huggingface.co/BytedanceDouyinContent/SAIL-VL-2B/},
    author = {Bytedance Douyin Content Team},
    month = {December},
    year = {2024}
}

Contributions

This work is conducted by Bytedance Douyin Content Team, authored by:

{Hongyuan Dong, Zijian Kang, Weijie Yin}, Xiao Liang, Chao Feng, Jiao Ran

{*} Equal Contributions.

We also appreciate the support from the model evaluation team:

Zirui Guo, Yan Qiu, Yaling Mou, Ming Jiang, Jingwei Sun

And from AI platform team:

Huiyu Yu, Lin Dong, Yong Zhang

License

This project is licensed under Apache License 2.0.

Contact

If you have any question, please feel free to contact us: [email protected]

Downloads last month
112
Safetensors
Model size
2.47B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for BytedanceDouyinContent/SAIL-VL-1d5-2B

Base model

Qwen/Qwen2.5-1.5B
Finetuned
Qwen/Qwen2.5-1.5B-Instruct
Finetuned
(708)
this model