---
base_model: google/gemma-2-9b-it
language:
- uk
library_name: peft
license: cc-by-nc-4.0
pipeline_tag: text-generation
---

# Empowering Smaller Models: Tuning Gemma for Ukrainian Exam Tasks

This model, presented in [Empowering Smaller Models: Tuning LLaMA and Gemma with Chain-of-Thought for Ukrainian Exam Tasks (arXiv:2503.13988)](https://arxiv.org/abs/2503.13988), is a 4-bit quantized PEFT adaptation of `google/gemma-2-9b-it` fine-tuned on Ukrainian language and literature exam tasks from the ZNO (EIE) & NMT datasets.  It's designed to generate the correct answer letter for multiple-choice questions.  The model takes a prompt consisting of the question and answer choices and outputs the letter corresponding to the correct answer.

## Model Details

This model was developed using the PEFT library for efficient parameter-efficient fine-tuning.

- **Model type:**  Causal Language Model
- **Language(s) (NLP):** Ukrainian (uk)
- **License:** CC-BY-NC-4.0
- **Finetuned from model:** `google/gemma-2-9b-it`

### Model Sources

- **Repository:** https://github.com/AndriyAntypenko/UKR-GEC-LLM
- **Paper:** [Empowering Smaller Models: Tuning LLaMA and Gemma with Chain-of-Thought for Ukrainian Exam Tasks (arXiv:2503.13988)](https://arxiv.org/abs/2503.13988)


## Uses

### Direct Use

The model can be used directly for generating the letter corresponding to the correct answer for Ukrainian language and literature exam questions, formatted as shown in the example below.

### Downstream Use

This model could be integrated into educational applications or question-answering systems focused on Ukrainian language and literature.


### Out-of-Scope Use

This model is specifically trained for Ukrainian exam tasks and should not be used for other tasks or languages.  Its performance on other domains is not guaranteed and may be unreliable.

## Bias, Risks, and Limitations

The model's performance is heavily dependent on the quality and characteristics of the training data (ZNO (EIE) & NMT datasets). Biases present in this data may be reflected in the model's output.  The model's accuracy is limited to the specific types of questions present in the training data.  Over-reliance on this model for high-stakes decisions without proper human oversight is strongly discouraged.

### Recommendations

Users should be cautious about the model's potential biases and limitations.  Human review of the model's output is crucial, especially for high-stakes applications.  Further research is needed to fully understand and mitigate potential biases.


## How to Get Started with the Model

```python
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
import torch

model_id = "ybelonogov/gemma-zno-eie" # Replace with the actual Hugging Face model ID

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True)

prompt = """<bos><start_of_turn>user
Дайте розгорнуту відповідь на завдання, починаючи з ключового слова "Відповідь:" та використовуючи лише наведені нижче варіанти.

Завдання: ... [Your Question Here] ...

Варіанти відповіді:
А – ...
Б – ...
В – ...
Г – ...
Д – ...<end_of_turn>"""

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
generation_config = GenerationConfig(
    temperature=0.1,
    max_new_tokens=512,
    do_sample=False
)

outputs = model.generate(inputs["input_ids"], **generation_config.to_dict())
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

```

## Training Details

### Training Data

[More information needed -  link to dataset card or description of ZNO (EIE) & NMT datasets]

### Training Procedure

The model was fine-tuned using the PEFT library's QLoRA technique.  Further details about the training hyperparameters are required.


#### Training Hyperparameters

- **Training regime:** 4-bit quantization


## Evaluation

[More information needed on evaluation metrics, datasets, etc.]


## Environmental Impact

[More information needed]


## Technical Specifications

### Model Architecture and Objective

[More information needed]


### Compute Infrastructure

[More information needed]


## Citation

[More information about citation details needed]

## Glossary

[Add glossary if needed]

## More Information

[Add more information if needed]

## Model Card Authors

[Add author information]

## Model Card Contact

[Add contact information]