Add the library_name and link to the paper
#2
by
nielsr
HF Staff
- opened
README.md
CHANGED
@@ -2,7 +2,9 @@
|
|
2 |
language:
|
3 |
- ar
|
4 |
- en
|
5 |
-
|
|
|
|
|
6 |
tags:
|
7 |
- Arabic
|
8 |
- English
|
@@ -10,11 +12,9 @@ tags:
|
|
10 |
- Decoder
|
11 |
- causal-lm
|
12 |
- jais-family
|
13 |
-
license: apache-2.0
|
14 |
-
pipeline_tag: text-generation
|
15 |
---
|
16 |
-
# Jais Family Model Card
|
17 |
|
|
|
18 |
|
19 |
The Jais family of models is a comprehensive series of bilingual English-Arabic large language models (LLMs). These models are optimized to excel in Arabic while having strong English capabilities. We release two variants of foundation models that include:
|
20 |
|
@@ -213,8 +213,10 @@ We conducted a comprehensive evaluation of Jais models focusing on both English
|
|
213 |
| jais-family-1p3b-chat | 42.7 | 42.2 | 30.1 | 33.6 | 40.6 | 34.1 | 41.2 | 43 | 63.6 | 69.3 | 44.9 | 31.6 | 28 | 45.6 | 50.4 |
|
214 |
| jais-family-590m-chat | 37.8 | 39.1 | 28 |29.5 | 33.1 | 30.8 | 36.4 | 30.3 | 57.8 | 57.2 | 40.5 | 25.9 | 26.8 | 44.5 | 49.3 |
|
215 |
|
|
|
216 |
|
217 |
-
|
|
|
218 |
| **Adapted Models** | Avg | ArabicMMLU*| MMLU | EXAMS*| LitQA*| agqa | agrc | Hellaswag | PIQA | BoolQA | Situated QA | ARC-C | OpenBookQA | TruthfulQA | CrowS-Pairs |
|
219 |
|--------------------------|-------|------------|-------|-------|-------|------|------|------------|------|--------|-------------|-------|------------|------------|-------------|
|
220 |
| jais-adapted-70b | 51.5 | 55.9 | 36.8 | 42.3 | 58.3 | 48.6 | 54 | 61.5 | 68.4 | 68.4 | 42.1 | 42.6 | 33 | 50.2 | 58.3 |
|
@@ -226,45 +228,6 @@ We conducted a comprehensive evaluation of Jais models focusing on both English
|
|
226 |
|
227 |
</div>
|
228 |
|
229 |
-
Arabic benchmarks are translated using an in-house MT model and reviewed by Arabic linguists. Benchmarks labeled with an asterisk (*) are natively Arabic; for further details, see the [Jais paper](https://arxiv.org/abs/2308.16149). Additionally, we include [ArabicMMLU](https://arxiv.org/abs/2402.12840), a native Arabic benchmark based on regional knowledge.
|
230 |
-
|
231 |
-
|
232 |
-
### English evaluation results:
|
233 |
-
|
234 |
-
<div class="table-container">
|
235 |
-
|
236 |
-
| **Models** | Avg | MMLU | RACE | Hellaswag | PIQA | BoolQA | SIQA | ARC-Challenge | OpenBookQA | Winogrande | TruthfulQA | CrowS-Pairs |
|
237 |
-
|--------------------------|----------|------|------|-----------|------|--------|------|---------------|------------|------------|----------------|-------------|
|
238 |
-
| jais-family-30b-16k | 59.3 | 42.2 | 40.5 | 79.7 | 80.6 | 78.7 | 48.8 | 50.3 | 44.2 | 71.6 | 43.5 | 72.6 |
|
239 |
-
| jais-family-30b-8k | 58.8 | 42.3 | 40.3 | 79.1 | 80.5 | 80.9 | 49.3 | 48.4 | 43.2 | 70.6 | 40.3 | 72.3 |
|
240 |
-
| jais-family-13b | 54.6 | 32.3 | 39 | 72 | 77.4 | 73.9 | 47.9 | 43.2 | 40 | 67.1 | 36.1 | 71.7 |
|
241 |
-
| jais-family-6p7b | 53.1 | 32 | 38 | 69.3 | 76 | 71.7 | 47.1 | 40.3 | 37.4 | 65.1 | 34.4 | 72.5 |
|
242 |
-
| jais-family-2p7b | 51 | 29.4 | 38 | 62.7 | 74.1 | 67.4 | 45.6 | 35.1 | 35.6 | 62.9 | 40.1 | 70.2 |
|
243 |
-
| jais-family-1p3b | 48.7 | 28.2 | 35.4 | 55.4 | 72 | 62.7 | 44.9 | 30.7 | 36.2 | 60.9 | 40.4 | 69 |
|
244 |
-
| jais-family-590m | 45.2 | 27.8 | 32.9 | 46.1 | 68.1 | 60.4 | 43.2 | 25.6 | 30.8 | 55.8 | 40.9 | 65.3 |
|
245 |
-
| jais-family-30b-16k-chat | 58.8 | 42 | 41.1 | 76.2 | 73.3 | 84.6 | 60.3 | 48.4 | 40.8 | 68.2 | 44.8 | 67 |
|
246 |
-
| jais-family-30b-8k-chat | 60.3 | 40.6 | 47.1 | 78.9 | 72.7 | 90.6 | 60 | 50.1 | 43.2 | 70.6 | 44.9 | 64.2 |
|
247 |
-
| jais-family-13b-chat | 57.5 | 36.6 | 42.6 | 75 | 75.8 | 87.6 | 54.4 | 47.9 | 42 | 65 | 40.6 | 64.5 |
|
248 |
-
| jais-family-6p7b-chat | 56 | 36.6 | 41.3 | 72 | 74 | 86.9 | 55.4 | 44.6 | 40 | 62.4 | 41 | 62.2 |
|
249 |
-
| jais-family-2p7b-chat | 52.8 | 32.7 | 40.4 | 62.2 | 71 | 84.1 | 54 | 37.2 | 36.8 | 61.4 | 40.9 | 59.8 |
|
250 |
-
| jais-family-1p3b-chat | 49.3 | 31.9 | 37.4 | 54.5 | 70.2 | 77.8 | 49.8 | 34.4 | 35.6 | 52.7 | 37.2 | 60.8 |
|
251 |
-
| jais-family-590m-chat | 42.6 | 27.9 | 33.4 | 33.1 | 63.7 | 60.1 | 45.3 | 26.7 | 25.8 | 50.5 | 44.5 | 57.7 |
|
252 |
-
|
253 |
-
</div>
|
254 |
-
|
255 |
-
<div class="table-container">
|
256 |
-
|
257 |
-
|**Adapted Models**| Avg | MMLU | RACE | Hellaswag | PIQA | BoolQA | SIQA | ARC-Challenge | OpenBookQA | Winogrande | TruthfulQA | CrowS-Pairs |
|
258 |
-
|--------------------------|----------|------|------|-----------|------|--------|------|---------------|------------|------------|----------------|-------------|
|
259 |
-
| jais-adapted-70b | 60.1 | 40.4 | 38.5 | 81.2 | 81.1 | 81.2 | 48.1 | 50.4 | 45 | 75.8 | 45.7 | 74 |
|
260 |
-
| jais-adapted-13b | 56 | 33.8 | 39.5 | 76.5 | 78.6 | 77.8 | 44.6 | 45.9 | 44.4 | 71.4 | 34.6 | 69 |
|
261 |
-
| jais-adapted-7b | 55.7 | 32.2 | 39.8 | 75.3 | 78.8 | 75.7 | 45.2 | 42.8 | 43 | 68 | 38.3 | 73.1 |
|
262 |
-
| jais-adapted-70b-chat | 61.4 | 38.7 | 42.9 | 82.7 | 81.2 | 89.6 | 52.9 | 54.9 | 44.4 | 75.7 | 44 | 68.8 |
|
263 |
-
| jais-adapted-13b-chat | 58.5 | 34.9 | 42.4 | 79.6 | 79.7 | 88.2 | 50.5 | 48.5 | 42.4 | 70.3 | 42.2 | 65.1 |
|
264 |
-
| jais-adapted-7b-chat | 58.5 | 33.8 | 43.9 | 77.8 | 79.4 | 87.1 | 47.3 | 46.9 | 43.4 | 69.9 | 42 | 72.4 |
|
265 |
-
|
266 |
-
</div>
|
267 |
-
|
268 |
|
269 |
### GPT-4 evaluation
|
270 |
|
@@ -356,24 +319,4 @@ Unless required by applicable law or agreed to in writing, JAIS is distributed o
|
|
356 |
|
357 |
We release the Jais family of Arabic and English bilingual models. The wide range of pre-trained model sizes, the recipe for adapting English-centric models to Arabic, and the fine-tuning of all sizes unlocks numerous use cases commercially and academically in the Arabic setting.
|
358 |
|
359 |
-
Through this release, we aim to make LLMs more accessible to Arabic NLP researchers and companies, offering native Arabic models that provide better cultural understanding than English centric ones. The strategies we employ for pre-training, fine-tuning and adaptation to Arabic are extensible to other low and medium resource languages, paving the way for language-focused
|
360 |
-
|
361 |
-
#### Citation info
|
362 |
-
|
363 |
-
```bibtex
|
364 |
-
@misc{sengupta2023jais,
|
365 |
-
title={Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models},
|
366 |
-
author={Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin and Eric Xing},
|
367 |
-
year={2023},
|
368 |
-
eprint={2308.16149},
|
369 |
-
archivePrefix={arXiv},
|
370 |
-
primaryClass={cs.CL}
|
371 |
-
}
|
372 |
-
|
373 |
-
@article{jaisfamilymodelcard,
|
374 |
-
title={Jais Family Model Card},
|
375 |
-
author={Inception},
|
376 |
-
year={2024},
|
377 |
-
url = {https://huggingface.co/inceptionai/jais-family-30b-16k-chat/blob/main/README.md}
|
378 |
-
}
|
379 |
-
```
|
|
|
2 |
language:
|
3 |
- ar
|
4 |
- en
|
5 |
+
license: apache-2.0
|
6 |
+
library_name: transformers
|
7 |
+
pipeline_tag: text-generation
|
8 |
tags:
|
9 |
- Arabic
|
10 |
- English
|
|
|
12 |
- Decoder
|
13 |
- causal-lm
|
14 |
- jais-family
|
|
|
|
|
15 |
---
|
|
|
16 |
|
17 |
+
# Jais Family Model Card
|
18 |
|
19 |
The Jais family of models is a comprehensive series of bilingual English-Arabic large language models (LLMs). These models are optimized to excel in Arabic while having strong English capabilities. We release two variants of foundation models that include:
|
20 |
|
|
|
213 |
| jais-family-1p3b-chat | 42.7 | 42.2 | 30.1 | 33.6 | 40.6 | 34.1 | 41.2 | 43 | 63.6 | 69.3 | 44.9 | 31.6 | 28 | 45.6 | 50.4 |
|
214 |
| jais-family-590m-chat | 37.8 | 39.1 | 28 |29.5 | 33.1 | 30.8 | 36.4 | 30.3 | 57.8 | 57.2 | 40.5 | 25.9 | 26.8 | 44.5 | 49.3 |
|
215 |
|
216 |
+
</div>
|
217 |
|
218 |
+
<div class="table-container">
|
219 |
+
|
220 |
| **Adapted Models** | Avg | ArabicMMLU*| MMLU | EXAMS*| LitQA*| agqa | agrc | Hellaswag | PIQA | BoolQA | Situated QA | ARC-C | OpenBookQA | TruthfulQA | CrowS-Pairs |
|
221 |
|--------------------------|-------|------------|-------|-------|-------|------|------|------------|------|--------|-------------|-------|------------|------------|-------------|
|
222 |
| jais-adapted-70b | 51.5 | 55.9 | 36.8 | 42.3 | 58.3 | 48.6 | 54 | 61.5 | 68.4 | 68.4 | 42.1 | 42.6 | 33 | 50.2 | 58.3 |
|
|
|
228 |
|
229 |
</div>
|
230 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
231 |
|
232 |
### GPT-4 evaluation
|
233 |
|
|
|
319 |
|
320 |
We release the Jais family of Arabic and English bilingual models. The wide range of pre-trained model sizes, the recipe for adapting English-centric models to Arabic, and the fine-tuning of all sizes unlocks numerous use cases commercially and academically in the Arabic setting.
|
321 |
|
322 |
+
Through this release, we aim to make LLMs more accessible to Arabic NLP researchers and companies, offering native Arabic models that provide better cultural understanding than English centric ones. The strategies we employ for pre-training, fine-tuning and adaptation to Arabic are extensible to other low and medium resource languages, paving the way for language-focused
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|