aashish1904 commited on
Commit
f70a09f
·
verified ·
1 Parent(s): ec1494c

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +542 -0
README.md ADDED
@@ -0,0 +1,542 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ ---
3
+
4
+ license: gemma
5
+ tags:
6
+ - gemma3
7
+ - gemma
8
+ - google
9
+ pipeline_tag: text-generation
10
+ library_name: transformers
11
+ extra_gated_heading: Access Gemma on Hugging Face
12
+ extra_gated_prompt: >-
13
+ To access Gemma on Hugging Face, you’re required to review and agree to
14
+ Google’s usage license. To do this, please ensure you’re logged in to Hugging
15
+ Face and click below. Requests are processed immediately.
16
+ extra_gated_button_content: Acknowledge license
17
+
18
+ ---
19
+
20
+ [![QuantFactory Banner](https://lh7-rt.googleusercontent.com/docsz/AD_4nXeiuCm7c8lEwEJuRey9kiVZsRn2W-b4pWlu3-X534V3YmVuVc2ZL-NXg2RkzSOOS2JXGHutDuyyNAUtdJI65jGTo8jT9Y99tMi4H4MqL44Uc5QKG77B0d6-JfIkZHFaUA71-RtjyYZWVIhqsNZcx8-OMaA?key=xt3VSDoCbmTY7o-cwwOFwQ)](https://hf.co/QuantFactory)
21
+
22
+
23
+ # QuantFactory/gemma-3-270m-GGUF
24
+ This is quantized version of [google/gemma-3-270m](https://huggingface.co/google/gemma-3-270m) created using llama.cpp
25
+
26
+ # Original Model Card
27
+
28
+
29
+ # Gemma 3 model card
30
+
31
+ **Model Page**: [Gemma](https://ai.google.dev/gemma/docs/core)
32
+
33
+ **Resources and Technical Documentation**:
34
+
35
+ * [Gemma 3 Technical Report][g3-tech-report]
36
+ * [Responsible Generative AI Toolkit][rai-toolkit]
37
+ * [Gemma on Kaggle][kaggle-gemma]
38
+ * [Gemma on Vertex Model Garden][vertex-mg-gemma3]
39
+
40
+ **Terms of Use**: [Terms][terms]
41
+
42
+ **Authors**: Google DeepMind
43
+
44
+ ## Model Information
45
+
46
+ Summary description and brief definition of inputs and outputs.
47
+
48
+ ### Description
49
+
50
+ Gemma is a family of lightweight, state-of-the-art open models from Google,
51
+ built from the same research and technology used to create the Gemini models.
52
+ Gemma 3 models are multimodal, handling text and image input and generating text
53
+ output, with open weights for both pre-trained variants and instruction-tuned
54
+ variants. Gemma 3 has a large, 128K context window, multilingual support in over
55
+ 140 languages, and is available in more sizes than previous versions. Gemma 3
56
+ models are well-suited for a variety of text generation and image understanding
57
+ tasks, including question answering, summarization, and reasoning. Their
58
+ relatively small size makes it possible to deploy them in environments with
59
+ limited resources such as laptops, desktops or your own cloud infrastructure,
60
+ democratizing access to state of the art AI models and helping foster innovation
61
+ for everyone.
62
+
63
+ ### Inputs and outputs
64
+
65
+ - **Input:**
66
+ - Text string, such as a question, a prompt, or a document to be summarized
67
+ - Images, normalized to 896 x 896 resolution and encoded to 256 tokens
68
+ each, for the 4B, 12B, and 27B sizes.
69
+ - Total input context of 128K tokens for the 4B, 12B, and 27B sizes, and
70
+ 32K tokens for the 1B and 270M sizes.
71
+
72
+ - **Output:**
73
+ - Generated text in response to the input, such as an answer to a
74
+ question, analysis of image content, or a summary of a document
75
+ - Total output context up to 128K tokens for the 4B, 12B, and 27B sizes,
76
+ and 32K tokens for the 1B and 270M sizes per request, subtracting the
77
+ request input tokens
78
+
79
+ ### Citation
80
+
81
+ ```none
82
+ @article{gemma_2025,
83
+ title={Gemma 3},
84
+ url={https://arxiv.org/abs/2503.19786},
85
+ publisher={Google DeepMind},
86
+ author={Gemma Team},
87
+ year={2025}
88
+ }
89
+ ```
90
+
91
+ ## Model Data
92
+
93
+ Data used for model training and how the data was processed.
94
+
95
+ ### Training Dataset
96
+
97
+ These models were trained on a dataset of text data that includes a wide variety
98
+ of sources. The 27B model was trained with 14 trillion tokens, the 12B model was
99
+ trained with 12 trillion tokens, 4B model was trained with 4 trillion tokens,
100
+ the 1B with 2 trillion tokens, and the 270M with 6 trillion tokens. The
101
+ knowledge cutoff date for the training data was August 2024. Here are the key
102
+ components:
103
+
104
+ - Web Documents: A diverse collection of web text ensures the model is
105
+ exposed to a broad range of linguistic styles, topics, and vocabulary. The
106
+ training dataset includes content in over 140 languages.
107
+ - Code: Exposing the model to code helps it to learn the syntax and
108
+ patterns of programming languages, which improves its ability to generate
109
+ code and understand code-related questions.
110
+ - Mathematics: Training on mathematical text helps the model learn logical
111
+ reasoning, symbolic representation, and to address mathematical queries.
112
+ - Images: A wide range of images enables the model to perform image
113
+ analysis and visual data extraction tasks.
114
+
115
+ The combination of these diverse data sources is crucial for training a powerful
116
+ multimodal model that can handle a wide variety of different tasks and data
117
+ formats.
118
+
119
+ ### Data Preprocessing
120
+
121
+ Here are the key data cleaning and filtering methods applied to the training
122
+ data:
123
+
124
+ - CSAM Filtering: Rigorous CSAM (Child Sexual Abuse Material) filtering
125
+ was applied at multiple stages in the data preparation process to ensure
126
+ the exclusion of harmful and illegal content.
127
+ - Sensitive Data Filtering: As part of making Gemma pre-trained models
128
+ safe and reliable, automated techniques were used to filter out certain
129
+ personal information and other sensitive data from training sets.
130
+ - Additional methods: Filtering based on content quality and safety in
131
+ line with [our policies][safety-policies].
132
+
133
+ ## Implementation Information
134
+
135
+ Details about the model internals.
136
+
137
+ ### Hardware
138
+
139
+ Gemma was trained using [Tensor Processing Unit (TPU)][tpu] hardware (TPUv4p,
140
+ TPUv5p and TPUv5e). Training vision-language models (VLMS) requires significant
141
+ computational power. TPUs, designed specifically for matrix operations common in
142
+ machine learning, offer several advantages in this domain:
143
+
144
+ - Performance: TPUs are specifically designed to handle the massive
145
+ computations involved in training VLMs. They can speed up training
146
+ considerably compared to CPUs.
147
+ - Memory: TPUs often come with large amounts of high-bandwidth memory,
148
+ allowing for the handling of large models and batch sizes during training.
149
+ This can lead to better model quality.
150
+ - Scalability: TPU Pods (large clusters of TPUs) provide a scalable
151
+ solution for handling the growing complexity of large foundation models.
152
+ You can distribute training across multiple TPU devices for faster and more
153
+ efficient processing.
154
+ - Cost-effectiveness: In many scenarios, TPUs can provide a more
155
+ cost-effective solution for training large models compared to CPU-based
156
+ infrastructure, especially when considering the time and resources saved
157
+ due to faster training.
158
+ - These advantages are aligned with
159
+ [Google's commitments to operate sustainably][sustainability].
160
+
161
+ ### Software
162
+
163
+ Training was done using [JAX][jax] and [ML Pathways][ml-pathways].
164
+
165
+ JAX allows researchers to take advantage of the latest generation of hardware,
166
+ including TPUs, for faster and more efficient training of large models. ML
167
+ Pathways is Google's latest effort to build artificially intelligent systems
168
+ capable of generalizing across multiple tasks. This is specially suitable for
169
+ foundation models, including large language models like these ones.
170
+
171
+ Together, JAX and ML Pathways are used as described in the
172
+ [paper about the Gemini family of models][gemini-2-paper]; *"the 'single
173
+ controller' programming model of Jax and Pathways allows a single Python
174
+ process to orchestrate the entire training run, dramatically simplifying the
175
+ development workflow."*
176
+
177
+ ## Evaluation
178
+
179
+ Model evaluation metrics and results.
180
+
181
+ ### Benchmark Results
182
+
183
+ These models were evaluated against a large collection of different datasets and
184
+ metrics to cover different aspects of text generation. Evaluation results marked
185
+ with **IT** are for instruction-tuned models. Evaluation results marked with
186
+ **PT** are for pre-trained models.
187
+
188
+ #### Gemma 3 270M
189
+
190
+ | **Benchmark** | **n-shot** | **Gemma 3 PT 270M** |
191
+ | :------------------------ | :-----------: | ------------------: |
192
+ | [HellaSwag][hellaswag] | 10-shot | 40.9 |
193
+ | [BoolQ][boolq] | 0-shot | 61.4 |
194
+ | [PIQA][piqa] | 0-shot | 67.7 |
195
+ | [TriviaQA][triviaqa] | 5-shot | 15.4 |
196
+ | [ARC-c][arc] | 25-shot | 29.0 |
197
+ | [ARC-e][arc] | 0-shot | 57.7 |
198
+ | [WinoGrande][winogrande] | 5-shot | 52.0 |
199
+
200
+ [hellaswag]: https://arxiv.org/abs/1905.07830
201
+ [boolq]: https://arxiv.org/abs/1905.10044
202
+ [piqa]: https://arxiv.org/abs/1911.11641
203
+ [triviaqa]: https://arxiv.org/abs/1705.03551
204
+ [arc]: https://arxiv.org/abs/1911.01547
205
+ [winogrande]: https://arxiv.org/abs/1907.10641
206
+
207
+ | **Benchmark** | **n-shot** | **Gemma 3 IT 270m** |
208
+ | :------------------------ | :-----------: | ------------------: |
209
+ | [HellaSwag][hellaswag] | 0-shot | 37.7 |
210
+ | [PIQA][piqa] | 0-shot | 66.2 |
211
+ | [ARC-c][arc] | 0-shot | 28.2 |
212
+ | [WinoGrande][winogrande] | 0-shot | 52.3 |
213
+ | [BIG-Bench Hard][bbh] | few-shot | 26.7 |
214
+ | [IF Eval][ifeval] | 0-shot | 51.2 |
215
+
216
+ [hellaswag]: https://arxiv.org/abs/1905.07830
217
+ [piqa]: https://arxiv.org/abs/1911.11641
218
+ [arc]: https://arxiv.org/abs/1911.01547
219
+ [winogrande]: https://arxiv.org/abs/1907.10641
220
+ [bbh]: https://paperswithcode.com/dataset/bbh
221
+ [bbh]: https://paperswithcode.com/dataset/bbh
222
+ [ifeval]: https://arxiv.org/abs/2311.07911
223
+
224
+ #### Gemma 3 1B, 4B, 12B & 27B
225
+
226
+ ##### Reasoning and factuality
227
+
228
+ | Benchmark | n-shot | Gemma 3 IT 1B | Gemma 3 IT 4B | Gemma 3 IT 12B | Gemma 3 IT 27B |
229
+ |--------------------------------|--------|:-------------:|:-------------:|:--------------:|:--------------:|
230
+ | [GPQA][gpqa] Diamond | 0-shot | 19.2 | 30.8 | 40.9 | 42.4 |
231
+ | [SimpleQA][simpleqa] | 0-shot | 2.2 | 4.0 | 6.3 | 10.0 |
232
+ | [FACTS Grounding][facts-grdg] | - | 36.4 | 70.1 | 75.8 | 74.9 |
233
+ | [BIG-Bench Hard][bbh] | 0-shot | 39.1 | 72.2 | 85.7 | 87.6 |
234
+ | [BIG-Bench Extra Hard][bbeh] | 0-shot | 7.2 | 11.0 | 16.3 | 19.3 |
235
+ | [IFEval][ifeval] | 0-shot | 80.2 | 90.2 | 88.9 | 90.4 |
236
+
237
+ | Benchmark | n-shot | Gemma 3 PT 1B | Gemma 3 PT 4B | Gemma 3 PT 12B | Gemma 3 PT 27B |
238
+ | ------------------------------ |----------|:--------------:|:-------------:|:--------------:|:--------------:|
239
+ | [HellaSwag][hellaswag] | 10-shot | 62.3 | 77.2 | 84.2 | 85.6 |
240
+ | [BoolQ][boolq] | 0-shot | 63.2 | 72.3 | 78.8 | 82.4 |
241
+ | [PIQA][piqa] | 0-shot | 73.8 | 79.6 | 81.8 | 83.3 |
242
+ | [SocialIQA][socialiqa] | 0-shot | 48.9 | 51.9 | 53.4 | 54.9 |
243
+ | [TriviaQA][triviaqa] | 5-shot | 39.8 | 65.8 | 78.2 | 85.5 |
244
+ | [Natural Questions][naturalq] | 5-shot | 9.48 | 20.0 | 31.4 | 36.1 |
245
+ | [ARC-c][arc] | 25-shot | 38.4 | 56.2 | 68.9 | 70.6 |
246
+ | [ARC-e][arc] | 0-shot | 73.0 | 82.4 | 88.3 | 89.0 |
247
+ | [WinoGrande][winogrande] | 5-shot | 58.2 | 64.7 | 74.3 | 78.8 |
248
+ | [BIG-Bench Hard][bbh] | few-shot | 28.4 | 50.9 | 72.6 | 77.7 |
249
+ | [DROP][drop] | 1-shot | 42.4 | 60.1 | 72.2 | 77.2 |
250
+
251
+ [gpqa]: https://arxiv.org/abs/2311.12022
252
+ [simpleqa]: https://arxiv.org/abs/2411.04368
253
+ [facts-grdg]: https://goo.gle/FACTS_paper
254
+ [bbeh]: https://github.com/google-deepmind/bbeh
255
+ [ifeval]: https://arxiv.org/abs/2311.07911
256
+ [hellaswag]: https://arxiv.org/abs/1905.07830
257
+ [boolq]: https://arxiv.org/abs/1905.10044
258
+ [piqa]: https://arxiv.org/abs/1911.11641
259
+ [socialiqa]: https://arxiv.org/abs/1904.09728
260
+ [triviaqa]: https://arxiv.org/abs/1705.03551
261
+ [naturalq]: https://github.com/google-research-datasets/natural-questions
262
+ [arc]: https://arxiv.org/abs/1911.01547
263
+ [winogrande]: https://arxiv.org/abs/1907.10641
264
+ [bbh]: https://paperswithcode.com/dataset/bbh
265
+ [drop]: https://arxiv.org/abs/1903.00161
266
+
267
+ ##### STEM and code
268
+
269
+ | Benchmark | n-shot | Gemma 3 IT 1B | Gemma 3 IT 4B | Gemma 3 IT 12B | Gemma 3 IT 27B |
270
+ |----------------------------|--------|:-------------:|:-------------:|:--------------:|:--------------:|
271
+ | [MMLU][mmlu] (Pro) | 0-shot | 14.7 | 43.6 | 60.6 | 67.5 |
272
+ | [LiveCodeBench][lcb] | 0-shot | 1.9 | 12.6 | 24.6 | 29.7 |
273
+ | [Bird-SQL][bird-sql] (dev) | - | 6.4 | 36.3 | 47.9 | 54.4 |
274
+ | [Math][math] | 0-shot | 48.0 | 75.6 | 83.8 | 89.0 |
275
+ | HiddenMath | 0-shot | 15.8 | 43.0 | 54.5 | 60.3 |
276
+ | [MBPP][mbpp] | 3-shot | 35.2 | 63.2 | 73.0 | 74.4 |
277
+ | [HumanEval][humaneval] | 0-shot | 41.5 | 71.3 | 85.4 | 87.8 |
278
+ | [Natural2Code][nat2code] | 0-shot | 56.0 | 70.3 | 80.7 | 84.5 |
279
+ | [GSM8K][gsm8k] | 0-shot | 62.8 | 89.2 | 94.4 | 95.9 |
280
+
281
+ | Benchmark | n-shot | Gemma 3 PT 4B | Gemma 3 PT 12B | Gemma 3 PT 27B |
282
+ | ------------------------------ |----------------|:-------------:|:--------------:|:--------------:|
283
+ | [MMLU][mmlu] | 5-shot | 59.6 | 74.5 | 78.6 |
284
+ | [MMLU][mmlu] (Pro COT) | 5-shot | 29.2 | 45.3 | 52.2 |
285
+ | [AGIEval][agieval] | 3-5-shot | 42.1 | 57.4 | 66.2 |
286
+ | [MATH][math] | 4-shot | 24.2 | 43.3 | 50.0 |
287
+ | [GSM8K][gsm8k] | 8-shot | 38.4 | 71.0 | 82.6 |
288
+ | [GPQA][gpqa] | 5-shot | 15.0 | 25.4 | 24.3 |
289
+ | [MBPP][mbpp] | 3-shot | 46.0 | 60.4 | 65.6 |
290
+ | [HumanEval][humaneval] | 0-shot | 36.0 | 45.7 | 48.8 |
291
+
292
+ [mmlu]: https://arxiv.org/abs/2009.03300
293
+ [agieval]: https://arxiv.org/abs/2304.06364
294
+ [math]: https://arxiv.org/abs/2103.03874
295
+ [gsm8k]: https://arxiv.org/abs/2110.14168
296
+ [gpqa]: https://arxiv.org/abs/2311.12022
297
+ [mbpp]: https://arxiv.org/abs/2108.07732
298
+ [humaneval]: https://arxiv.org/abs/2107.03374
299
+ [lcb]: https://arxiv.org/abs/2403.07974
300
+ [bird-sql]: https://arxiv.org/abs/2305.03111
301
+ [nat2code]: https://arxiv.org/abs/2405.04520
302
+
303
+ #### Multilingual
304
+
305
+ | Benchmark | n-shot | Gemma 3 IT 1B | Gemma 3 IT 4B | Gemma 3 IT 12B | Gemma 3 IT 27B |
306
+ |--------------------------------------|--------|:-------------:|:-------------:|:--------------:|:--------------:|
307
+ | [Global-MMLU-Lite][global-mmlu-lite] | 0-shot | 34.2 | 54.5 | 69.5 | 75.1 |
308
+ | [ECLeKTic][eclektic] | 0-shot | 1.4 | 4.6 | 10.3 | 16.7 |
309
+ | [WMT24++][wmt24pp] | 0-shot | 35.9 | 46.8 | 51.6 | 53.4 |
310
+
311
+ | Benchmark | Gemma 3 PT 1B | Gemma 3 PT 4B | Gemma 3 PT 12B | Gemma 3 PT 27B |
312
+ | ------------------------------------ |:-------------:|:-------------:|:--------------:|:--------------:|
313
+ | [MGSM][mgsm] | 2.04 | 34.7 | 64.3 | 74.3 |
314
+ | [Global-MMLU-Lite][global-mmlu-lite] | 24.9 | 57.0 | 69.4 | 75.7 |
315
+ | [WMT24++][wmt24pp] (ChrF) | 36.7 | 48.4 | 53.9 | 55.7 |
316
+ | [FloRes][flores] | 29.5 | 39.2 | 46.0 | 48.8 |
317
+ | [XQuAD][xquad] (all) | 43.9 | 68.0 | 74.5 | 76.8 |
318
+ | [ECLeKTic][eclektic] | 4.69 | 11.0 | 17.2 | 24.4 |
319
+ | [IndicGenBench][indicgenbench] | 41.4 | 57.2 | 61.7 | 63.4 |
320
+
321
+ [mgsm]: https://arxiv.org/abs/2210.03057
322
+ [flores]: https://arxiv.org/abs/2106.03193
323
+ [xquad]: https://arxiv.org/abs/1910.11856v3
324
+ [global-mmlu-lite]: https://huggingface.co/datasets/CohereForAI/Global-MMLU-Lite
325
+ [wmt24pp]: https://arxiv.org/abs/2502.12404v1
326
+ [eclektic]: https://arxiv.org/abs/2502.21228
327
+ [indicgenbench]: https://arxiv.org/abs/2404.16816
328
+
329
+ ##### Multimodal
330
+
331
+ | Benchmark | Gemma 3 IT 4B | Gemma 3 IT 12B | Gemma 3 IT 27B |
332
+ |-----------------------------------|:-------------:|:--------------:|:--------------:|
333
+ | [MMMU][mmmu] (val) | 48.8 | 59.6 | 64.9 |
334
+ | [DocVQA][docvqa] | 75.8 | 87.1 | 86.6 |
335
+ | [InfoVQA][info-vqa] | 50.0 | 64.9 | 70.6 |
336
+ | [TextVQA][textvqa] | 57.8 | 67.7 | 65.1 |
337
+ | [AI2D][ai2d] | 74.8 | 84.2 | 84.5 |
338
+ | [ChartQA][chartqa] | 68.8 | 75.7 | 78.0 |
339
+ | [VQAv2][vqav2] (val) | 62.4 | 71.6 | 71.0 |
340
+ | [MathVista][mathvista] (testmini) | 50.0 | 62.9 | 67.6 |
341
+
342
+ | Benchmark | Gemma 3 PT 4B | Gemma 3 PT 12B | Gemma 3 PT 27B |
343
+ | ------------------------------ |:-------------:|:--------------:|:--------------:|
344
+ | [COCOcap][coco-cap] | 102 | 111 | 116 |
345
+ | [DocVQA][docvqa] (val) | 72.8 | 82.3 | 85.6 |
346
+ | [InfoVQA][info-vqa] (val) | 44.1 | 54.8 | 59.4 |
347
+ | [MMMU][mmmu] (pt) | 39.2 | 50.3 | 56.1 |
348
+ | [TextVQA][textvqa] (val) | 58.9 | 66.5 | 68.6 |
349
+ | [RealWorldQA][realworldqa] | 45.5 | 52.2 | 53.9 |
350
+ | [ReMI][remi] | 27.3 | 38.5 | 44.8 |
351
+ | [AI2D][ai2d] | 63.2 | 75.2 | 79.0 |
352
+ | [ChartQA][chartqa] | 63.6 | 74.7 | 76.3 |
353
+ | [VQAv2][vqav2] | 63.9 | 71.2 | 72.9 |
354
+ | [BLINK][blinkvqa] | 38.0 | 35.9 | 39.6 |
355
+ | [OKVQA][okvqa] | 51.0 | 58.7 | 60.2 |
356
+ | [TallyQA][tallyqa] | 42.5 | 51.8 | 54.3 |
357
+ | [SpatialSense VQA][ss-vqa] | 50.9 | 60.0 | 59.4 |
358
+ | [CountBenchQA][countbenchqa] | 26.1 | 17.8 | 68.0 |
359
+
360
+ [coco-cap]: https://cocodataset.org/#home
361
+ [docvqa]: https://www.docvqa.org/
362
+ [info-vqa]: https://arxiv.org/abs/2104.12756
363
+ [mmmu]: https://arxiv.org/abs/2311.16502
364
+ [textvqa]: https://textvqa.org/
365
+ [realworldqa]: https://paperswithcode.com/dataset/realworldqa
366
+ [remi]: https://arxiv.org/html/2406.09175v1
367
+ [ai2d]: https://allenai.org/data/diagrams
368
+ [chartqa]: https://arxiv.org/abs/2203.10244
369
+ [vqav2]: https://visualqa.org/index.html
370
+ [blinkvqa]: https://arxiv.org/abs/2404.12390
371
+ [okvqa]: https://okvqa.allenai.org/
372
+ [tallyqa]: https://arxiv.org/abs/1810.12440
373
+ [ss-vqa]: https://arxiv.org/abs/1908.02660
374
+ [countbenchqa]: https://github.com/google-research/big_vision/blob/main/big_vision/datasets/countbenchqa/
375
+ [mathvista]: https://arxiv.org/abs/2310.02255
376
+
377
+ ## Ethics and Safety
378
+
379
+ Ethics and safety evaluation approach and results.
380
+
381
+ ### Evaluation Approach
382
+
383
+ Our evaluation methods include structured evaluations and internal red-teaming
384
+ testing of relevant content policies. Red-teaming was conducted by a number of
385
+ different teams, each with different goals and human evaluation metrics. These
386
+ models were evaluated against a number of different categories relevant to
387
+ ethics and safety, including:
388
+
389
+ - **Child Safety**: Evaluation of text-to-text and image to text prompts
390
+ covering child safety policies, including child sexual abuse and
391
+ exploitation.
392
+ - **Content Safety:** Evaluation of text-to-text and image to text prompts
393
+ covering safety policies including, harassment, violence and gore, and hate
394
+ speech.
395
+ - **Representational Harms**: Evaluation of text-to-text and image to text
396
+ prompts covering safety policies including bias, stereotyping, and harmful
397
+ associations or inaccuracies.
398
+
399
+ In addition to development level evaluations, we conduct "assurance
400
+ evaluations" which are our 'arms-length' internal evaluations for responsibility
401
+ governance decision making. They are conducted separately from the model
402
+ development team, to inform decision making about release. High level findings
403
+ are fed back to the model team, but prompt sets are held-out to prevent
404
+ overfitting and preserve the results' ability to inform decision making.
405
+ Assurance evaluation results are reported to our Responsibility & Safety Council
406
+ as part of release review.
407
+
408
+ ### Evaluation Results
409
+
410
+ For all areas of safety testing, we saw major improvements in the categories of
411
+ child safety, content safety, and representational harms relative to previous
412
+ Gemma models. All testing was conducted without safety filters to evaluate the
413
+ model capabilities and behaviors. For both text-to-text and image-to-text, and
414
+ across all model sizes, the model produced minimal policy violations, and showed
415
+ significant improvements over previous Gemma models' performance with respect
416
+ to ungrounded inferences. A limitation of our evaluations was they included only
417
+ English language prompts.
418
+
419
+ ## Usage and Limitations
420
+
421
+ These models have certain limitations that users should be aware of.
422
+
423
+ ### Intended Usage
424
+
425
+ Open vision-language models (VLMs) models have a wide range of applications
426
+ across various industries and domains. The following list of potential uses is
427
+ not comprehensive. The purpose of this list is to provide contextual information
428
+ about the possible use-cases that the model creators considered as part of model
429
+ training and development.
430
+
431
+ - Content Creation and Communication
432
+ - Text Generation: These models can be used to generate creative text
433
+ formats such as poems, scripts, code, marketing copy, and email drafts.
434
+ - Chatbots and Conversational AI: Power conversational interfaces
435
+ for customer service, virtual assistants, or interactive applications.
436
+ - Text Summarization: Generate concise summaries of a text corpus,
437
+ research papers, or reports.
438
+ - Image Data Extraction: These models can be used to extract,
439
+ interpret, and summarize visual data for text communications.
440
+ - Research and Education
441
+ - Natural Language Processing (NLP) and VLM Research: These
442
+ models can serve as a foundation for researchers to experiment with VLM
443
+ and NLP techniques, develop algorithms, and contribute to the
444
+ advancement of the field.
445
+ - Language Learning Tools: Support interactive language learning
446
+ experiences, aiding in grammar correction or providing writing practice.
447
+ - Knowledge Exploration: Assist researchers in exploring large
448
+ bodies of text by generating summaries or answering questions about
449
+ specific topics.
450
+
451
+ ### Limitations
452
+
453
+ - Training Data
454
+ - The quality and diversity of the training data significantly
455
+ influence the model's capabilities. Biases or gaps in the training data
456
+ can lead to limitations in the model's responses.
457
+ - The scope of the training dataset determines the subject areas
458
+ the model can handle effectively.
459
+ - Context and Task Complexity
460
+ - Models are better at tasks that can be framed with clear
461
+ prompts and instructions. Open-ended or highly complex tasks might be
462
+ challenging.
463
+ - A model's performance can be influenced by the amount of context
464
+ provided (longer context generally leads to better outputs, up to a
465
+ certain point).
466
+ - Language Ambiguity and Nuance
467
+ - Natural language is inherently complex. Models might struggle
468
+ to grasp subtle nuances, sarcasm, or figurative language.
469
+ - Factual Accuracy
470
+ - Models generate responses based on information they learned
471
+ from their training datasets, but they are not knowledge bases. They
472
+ may generate incorrect or outdated factual statements.
473
+ - Common Sense
474
+ - Models rely on statistical patterns in language. They might
475
+ lack the ability to apply common sense reasoning in certain situations.
476
+
477
+ ### Ethical Considerations and Risks
478
+
479
+ The development of vision-language models (VLMs) raises several ethical
480
+ concerns. In creating an open model, we have carefully considered the following:
481
+
482
+ - Bias and Fairness
483
+ - VLMs trained on large-scale, real-world text and image data can
484
+ reflect socio-cultural biases embedded in the training material. These
485
+ models underwent careful scrutiny, input data pre-processing described
486
+ and posterior evaluations reported in this card.
487
+ - Misinformation and Misuse
488
+ - VLMs can be misused to generate text that is false, misleading,
489
+ or harmful.
490
+ - Guidelines are provided for responsible use with the model, see the
491
+ [Responsible Generative AI Toolkit][rai-toolkit].
492
+ - Transparency and Accountability:
493
+ - This model card summarizes details on the models' architecture,
494
+ capabilities, limitations, and evaluation processes.
495
+ - A responsibly developed open model offers the opportunity to
496
+ share innovation by making VLM technology accessible to developers and
497
+ researchers across the AI ecosystem.
498
+
499
+ Risks identified and mitigations:
500
+
501
+ - **Perpetuation of biases**: It's encouraged to perform continuous
502
+ monitoring (using evaluation metrics, human review) and the exploration of
503
+ de-biasing techniques during model training, fine-tuning, and other use
504
+ cases.
505
+ - **Generation of harmful content**: Mechanisms and guidelines for content
506
+ safety are essential. Developers are encouraged to exercise caution and
507
+ implement appropriate content safety safeguards based on their specific
508
+ product policies and application use cases.
509
+ - **Misuse for malicious purposes**: Technical limitations and developer
510
+ and end-user education can help mitigate against malicious applications of
511
+ VLMs. Educational resources and reporting mechanisms for users to flag
512
+ misuse are provided. Prohibited uses of Gemma models are outlined in the
513
+ [Gemma Prohibited Use Policy][prohibited-use].
514
+ - **Privacy violations**: Models were trained on data filtered for removal
515
+ of certain personal information and other sensitive data. Developers are
516
+ encouraged to adhere to privacy regulations with privacy-preserving
517
+ techniques.
518
+
519
+ ### Benefits
520
+
521
+ At the time of release, this family of models provides high-performance open
522
+ vision-language model implementations designed from the ground up for
523
+ responsible AI development compared to similarly sized models.
524
+
525
+ Using the benchmark evaluation metrics described in this document, these models
526
+ have shown to provide superior performance to other, comparably-sized open model
527
+ alternatives.
528
+
529
+ [g3-tech-report]: https://arxiv.org/abs/2503.19786
530
+ [rai-toolkit]: https://ai.google.dev/responsible
531
+ [kaggle-gemma]: https://www.kaggle.com/models/google/gemma-3
532
+ [vertex-mg-gemma3]: https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/gemma3
533
+ [terms]: https://ai.google.dev/gemma/terms
534
+ [safety-policies]: https://ai.google/static/documents/ai-responsibility-update-published-february-2025.pdf
535
+ [prohibited-use]: https://ai.google.dev/gemma/prohibited_use_policy
536
+ [tpu]: https://cloud.google.com/tpu/docs/intro-to-tpu
537
+ [sustainability]: https://sustainability.google/operating-sustainably/
538
+ [jax]: https://github.com/jax-ml/jax
539
+ [ml-pathways]: https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/
540
+ [sustainability]: https://sustainability.google/operating-sustainably/
541
+ [gemini-2-paper]: https://arxiv.org/abs/2312.11805
542
+