Improve model card: Add pipeline tag, library name, code link, and sample usage
Browse filesThis PR enhances the model card by:
- Adding `pipeline_tag: image-text-to-text` to the metadata, which improves discoverability and categorization on the Hugging Face Hub.
- Adding `library_name: transformers` to the metadata, enabling the "Use in Transformers" widget and code snippets on the model page.
- Including a direct link to the project's GitHub repository for easier access to the associated code.
- Providing a clear, runnable Python code snippet using the `transformers` library for convenient model inference.
README.md
CHANGED
@@ -1,10 +1,12 @@
|
|
1 |
---
|
2 |
-
|
|
|
3 |
datasets:
|
4 |
- Code2Logic/GameQA-140K
|
5 |
- Code2Logic/GameQA-5K
|
6 |
-
|
7 |
-
|
|
|
8 |
---
|
9 |
|
10 |
***This model (GameQA-LLaVA-OV-7B) results from training LLaVA-OV-7B with GRPO solely on our [GameQA-5K](https://huggingface.co/datasets/Code2Logic/GameQA-5K) (sampled from the full [GameQA-140K](https://huggingface.co/datasets/Gabriel166/GameQA-140K) dataset).***
|
@@ -19,8 +21,48 @@ base_model:
|
|
19 |
|
20 |
This is the first work, to the best of our knowledge, that leverages ***game code*** to synthesize multimodal reasoning data for ***training*** VLMs. Furthermore, when trained with a GRPO strategy solely on **GameQA** (synthesized via our proposed **Code2Logic** approach), multiple cutting-edge open-source models exhibit significantly enhanced out-of-domain generalization.
|
21 |
|
22 |
-
[[π Paper](https://arxiv.org/abs/2505.13886)] [[π€ GameQA-140K Dataset](https://huggingface.co/datasets/Gabriel166/GameQA-140K)] [[π€ GameQA-5K Dataset](https://huggingface.co/datasets/Code2Logic/GameQA-5K)] [[π€ GameQA-InternVL3-8B](https://huggingface.co/Code2Logic/GameQA-InternVL3-8B) ] [[π€ GameQA-Qwen2.5-VL-7B](https://huggingface.co/Code2Logic/GameQA-Qwen2.5-VL-7B)] [[
|
23 |
|
24 |
## News
|
25 |
|
26 |
-
* We've open-sourced the ***three*** models trained with GRPO on GameQA on [Huggingface](https://huggingface.co/Code2Logic).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
+
base_model:
|
3 |
+
- llava-hf/llava-onevision-qwen2-7b-ov-hf
|
4 |
datasets:
|
5 |
- Code2Logic/GameQA-140K
|
6 |
- Code2Logic/GameQA-5K
|
7 |
+
license: apache-2.0
|
8 |
+
pipeline_tag: image-text-to-text
|
9 |
+
library_name: transformers
|
10 |
---
|
11 |
|
12 |
***This model (GameQA-LLaVA-OV-7B) results from training LLaVA-OV-7B with GRPO solely on our [GameQA-5K](https://huggingface.co/datasets/Code2Logic/GameQA-5K) (sampled from the full [GameQA-140K](https://huggingface.co/datasets/Gabriel166/GameQA-140K) dataset).***
|
|
|
21 |
|
22 |
This is the first work, to the best of our knowledge, that leverages ***game code*** to synthesize multimodal reasoning data for ***training*** VLMs. Furthermore, when trained with a GRPO strategy solely on **GameQA** (synthesized via our proposed **Code2Logic** approach), multiple cutting-edge open-source models exhibit significantly enhanced out-of-domain generalization.
|
23 |
|
24 |
+
[[π Paper](https://arxiv.org/abs/2505.13886)] [[\ud83d\udcbb Code](https://github.com/tongjingqi/Code2Logic)] [[π€ GameQA-140K Dataset](https://huggingface.co/datasets/Gabriel166/GameQA-140K)] [[π€ GameQA-5K Dataset](https://huggingface.co/datasets/Code2Logic/GameQA-5K)] [[π€ GameQA-InternVL3-8B](https://huggingface.co/Code2Logic/GameQA-InternVL3-8B) ] [[π€ GameQA-Qwen2.5-VL-7B](https://huggingface.co/Code2Logic/GameQA-Qwen2.5-VL-7B)] [[\ud83e\udd17 GameQA-LLaVA-OV-7B](https://huggingface.co/Code2Logic/GameQA-llava-onevision-qwen2-7b-ov-hf) ]
|
25 |
|
26 |
## News
|
27 |
|
28 |
+
* We've open-sourced the ***three*** models trained with GRPO on GameQA on [Huggingface](https://huggingface.co/Code2Logic).
|
29 |
+
|
30 |
+
## Usage
|
31 |
+
|
32 |
+
This model is compatible with the `transformers` library. Here's how to use it for image-to-text generation:
|
33 |
+
|
34 |
+
```python
|
35 |
+
import torch
|
36 |
+
from PIL import Image
|
37 |
+
from transformers import AutoProcessor, AutoModelForCausalLM
|
38 |
+
|
39 |
+
model_id = "Code2Logic/GameQA-llava-onevision-qwen2-7b-ov-hf"
|
40 |
+
|
41 |
+
# Load processor and model
|
42 |
+
processor = AutoProcessor.from_pretrained(model_id)
|
43 |
+
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).to("cuda")
|
44 |
+
|
45 |
+
# Load your image (replace with an actual image path or PIL Image object)
|
46 |
+
# Example: a screenshot of a GUI for a typical use case of this model
|
47 |
+
image = Image.open("your_gui_screenshot.jpg")
|
48 |
+
|
49 |
+
# Prepare your text prompt. The model is designed for multimodal tasks,
|
50 |
+
# so typical inputs involve both an image and a text query.
|
51 |
+
prompt = "What is highlighted in the screenshot? Provide a concise description."
|
52 |
+
|
53 |
+
# Construct the chat history format required by the model
|
54 |
+
messages = [
|
55 |
+
{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}
|
56 |
+
]
|
57 |
+
chat_prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
58 |
+
|
59 |
+
# Process inputs for the model
|
60 |
+
inputs = processor(text=chat_prompt, images=image, return_tensors="pt").to(model.device)
|
61 |
+
|
62 |
+
# Generate response
|
63 |
+
with torch.no_grad():
|
64 |
+
generated_ids = model.generate(**inputs, max_new_tokens=100) # Adjust max_new_tokens as needed
|
65 |
+
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
|
66 |
+
|
67 |
+
print(generated_text)
|
68 |
+
```
|