CLIP Model for Chart Understanding

This repository contains the CLIP model implementation from our paper "On the Perception Bottleneck of VLMs for Chart Understanding".

Overview

This CLIP model is specifically trained to address the perception bottleneck in Vision Language Models (VLMs) when processing and understanding charts and visualizations. Our work explores and aims to improve how CLIP effect its LVLMs.

Model Details

Model architecture: trained from openai/clip-vit-large-patch14-336
Training data: from our collected and synthetic hard negatives chart data(Vision4Chart Dataset)
Training method: NegCLIP Training

Citation

If you find this model useful in your research, please consider citing our paper:

@misc{liu2025perceptionbottleneckvlmschart,
      title={On the Perception Bottleneck of VLMs for Chart Understanding}, 
      author={Junteng Liu and Weihao Zeng and Xiwen Zhang and Yijun Wang and Zifei Shan and Junxian He},
      year={2025},
      eprint={2503.18435},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2503.18435}, 
}