GemMaroc
/

GemMaroc-4b-deita

@@ -1,6 +1,17 @@
 ---
 library_name: transformers
-tags: []
 ---
 # Model Card for Model ID
@@ -8,192 +19,133 @@ tags: []
 <!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
 library_name: transformers
+tags:
+- MoroccanArabic
+- Darija
+- GemMaroc
+datasets:
+- GemMaroc/TULU-3-50k-darija-english
+language:
+- ar
+- ary
+- en
+base_model:
+- google/gemma-3-27b-it
 ---
 # Model Card for Model ID
 <!-- Provide a quick summary of what the model is/does. -->
+# GemMaroc‑27B
+Unlocking **Moroccan Darija** proficiency in a state‑of‑the‑art large language model, trained with a *minimal‑data, green‑AI* recipe that preserves Gemma‑27B’s strong reasoning abilities while adding fluent Darija generation.
+---
+## Model at a glance
+|                     | Details                                                                                                                       |
+| ------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| **Model ID**        | `AbderrahmanSkiredj1/GemMaroc-27b-it`                                                                                         |
+| **Base model**      | [`google/gemma-3-27b`](https://huggingface.co/google/gemma-3-27b)                                                             |
+| **Architecture**    | Decoder‑only Transformer (Gemma 3)                                                                                            |
+| **Parameters**      | 27 billion                                                                                                                    |
+| **Context length**  | 2 048 tokens                                                                                                                  |
+| **Training regime** | Supervised fine‑tuning (LoRA → merged) on 50 K high‑quality Darija/English instructions TULU‑50K slice |
+| **Compute budget**  | 48 GPU·h (8 × H100‑80GB × 6 h) – ≈ 26 kWh / 10 kg CO₂e                                                                        |
+| **License**         | Apache 2.0                                                                                                                    |
+---
+## Why another Darija model?
+* **Inclusive AI** > 36 million speakers of Moroccan Arabic remain underserved by open LLMs.
+* **Quality‑over‑quantity** A carefully curated 50 K instruction set surfaces Darija competence without sacrificing cross‑lingual reasoning.
+* **Green AI** GemMaroc achieves Atlas‑Chat‑level Darija scores using < 2 % of the energy.
+---
+## Benchmark summary
+| Model            | Darija MMLU | Darija HellaSwag | GSM8K @5   | HellaSwag (EN) |
+| ---------------- | ----------- | ---------------- | ---------- | -------------- |
+| Atlas‑Chat‑27B   | **61.9 %**  | 48.4 %           | 82.0 %     | 77.8 %         |
+| **GemMaroc‑27B** | 61.6 %      | **60.5 %**       | **84.2 %** | **79.3 %**     |
+<sub>Zero‑shot accuracy; full table in the paper.</sub>
+---
+## Quick start
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
+model_id = "AbderrahmanSkiredj1/GemMaroc-27b-it"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model     = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype="auto",
+    device_map="auto"
+)
+pipe = pipeline(
+    "text-generation",
+    model=model,
+    tokenizer=tokenizer,
+    device_map="auto",
+    max_new_tokens=1024,
+    temperature=0.7,
+    repetition_penalty=1.2,
+    no_repeat_ngram_size=3,
+)
+messages = [
+    {"role": "user", "content": "شنو هي نظرية ‘butterfly effect’؟ فسّرها بدارجة ونقّط مثال بسيط."}
+]
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+print(pipe(prompt)[0]["generated_text"][len(prompt):])
+```
+### Chat template (Gemma 3 format)
+The tokenizer provides a baked‑in Jinja template that starts with a **begin‑of‑sequence** token (`<bos>`), then alternates user/model turns, each wrapped by `<start_of_turn>` … `<end_of_turn>` markers. When you set `add_generation_prompt=True` it ends after the opening model tag so the model can continue:
+```
+<bos><start_of_turn>user
+{user message}<end_of_turn>
+<start_of_turn>model
+```
+The assistant will keep generating tokens until it decides to emit `<end_of_turn>`.
+```python
+prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
+```
+No manual token juggling required—the call above handles BOS, turn delimiters, and newline placement automatically.
+---
+Pre‑quantised checkpoints will be published under the same repo tags (`gemmaroc‑27b‑awq‑int4`, `gemmaroc‑27b‑gguf‑q4_k_m`).
+---
+## Training recipe (one‑paragraph recap)
+1. **Data** Translate a 44 K reasoning slice of TULU 50K into Darija, keeping 20 % English for cross‑lingual robustness.
+2. **LoRA SFT** Rank 16, α = 32, 3 epochs, bf16, context 2 048.
+3. **Merge & push** Merge LoRA into base weights (`peft.merge_and_unload`), convert to safetensors, upload.
+---
+## Limitations & ethical considerations
+* Sentiment and abstractive summarisation still trail state‑of‑the‑art.
+* Tokeniser is unchanged; rare Darija spellings may fragment.
+* Model may inherit societal biases present in pre‑training data.
+* No RLHF / RLAIF safety alignment yet – apply a moderation layer in production.
+---
+## Citation
+If you use GemMaroc in your work, please cite:
+```bibtex
+@misc{abderrahman2025gemmaroc,
+  author       = {Skiredj, Abderrahman and Azhari, Ferdaous and Atou, Houdaifa and Tazi, Nouamane and Berrada, Ismail},
+  title        = {{GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data}},
+  year         = {2025},
+  publisher    = {Zenodo},
+  doi          = {10.5281/zenodo.15492072},
+  url          = {https://doi.org/10.5281/zenodo.15492072}
+}
+```