Create README.md

README.md

@@ -0,0 +1,93 @@
+## POINTS-Yi-1.5-9B-Chat
+
+### Introduction
+
+We are excited to announce the first version of POINTS, which integrates recent advancement in vision-language model and new techniques proposed by researchers from WeChat AI.
+
+### What's new in POINTS?
+
+**Key Innovations**
+
+1. **Strong Baseline**: We integrate the most recent advancement in vision-language model, i.e., CapFusion, Dual Vision Encoder, and
+Dynamic High Resolution, into POINTS.
+
+2. **Pre-training Dataset Filtering**: We propose to filter the pre-training dataset using perplexity as a metric. Utilizing this filtering strategy, we can significantly reduce the size of the pre-training dataset and improve the performance of the model.
+
+3. **Model Soup**: We propose to apply model soup to models, fine-tuned with different visual instruction tuning datasets, which can further significantly improve the performance of the model.
+
+<p align="center">
+    <img src="https://github.com/user-attachments/assets/6af35008-f501-400a-a870-b66a9bf2baab" width="60%"/>
+<p>
+
+
+### How to use POINTS?
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from transformers import CLIPImageProcessor
+from PIL import Image
+import torch
+
+
+image_path = '/path/to/local/image.jpg'
+prompt = 'please describe the image in detail'
+pil_image = Image.open(image_path)
+model_path = ''
+tokenizer = AutoTokenizer.from_pretrained(model_path)
+model = AutoModelForCausalLM.from_pretrained(
+    model_path, trust_remote_code=True, device_map='cuda').to(torch.bfloat16)
+image_processor = CLIPImageProcessor.from_pretrained(model_path)
+generation_config = {
+    'max_new_tokens': 1024,
+    'temperature': 0.0,
+    'top_p': 0.0,
+    'num_beams': 1,
+}
+res = model.chat(
+    pil_image,
+    prompt,
+    tokenizer,
+    image_processor,
+    True,
+    generation_config
+)
+print(res)
+```
+
+### Evaluation
+
+| Benchmark | InternVL2-8B | LLaVA-OneVision | POINTS |
+| :-------: | :----------: | :-------------: | :----: |
+| MMBench-dev-en | - | 80.8 | 83.2 |
+| MathVista | 58.3 | 62.3 | 60.7 |
+| HallucinationBench | 45.0 | 31.6 | 48.0 |
+| OCRBench | 79.4 | 62.2 | 70.6 | 
+| AI2D | 83.6 | 82.4 | 78.5 |
+| MMVet | 54.3 | 51.9 | 50.0 |
+| MMStar | 61.5 | 61.9 | 56.4 |
+| MMMU | 51.2 | 47.9 | 46.9 |
+| ScienceQA | 97.1 | 95.4 | 92.9 |
+| MME | 2215.1 | 1993.6 | 2017.8 |
+| RealWorldQA | 64.2 | 69.9 | 65.9 |
+| LLaVA-Wild | 73.3 | 81.0 | 69.3 |
+
+
+### Citation
+
+If you find our work helpful, feel free to cite us:
+
+```
+@article{liu2024points,
+  title={POINTS: Improving Your Vision-language Model with Affordable Strategies},
+  author={Liu, Yuan and Zhao, Zhongyin and Zhuang, Ziyuan and Tian, Le and Zhou, Xiao and Zhou, Jie},
+  journal={arXiv preprint arXiv:2409.04828},
+  year={2024}
+}
+
+@article{liu2024rethinking,
+  title={Rethinking Overlooked Aspects in Vision-Language Models},
+  author={Liu, Yuan and Tian, Le and Zhou, Xiao and Zhou, Jie},
+  journal={arXiv preprint arXiv:2405.11850},
+  year={2024}
+}
+```