|
|
--- |
|
|
language: |
|
|
- ms |
|
|
- en |
|
|
datasets: |
|
|
- mesolitica/Malaysian-Reasoning |
|
|
base_model: |
|
|
- mesolitica/Malaysian-Qwen2.5-1.5B-Instruct |
|
|
--- |
|
|
|
|
|
# Malaysian Qwen 2.5 1.5B Instruct Reasoning SFT |
|
|
|
|
|
Continue finetuning https://huggingface.co/mesolitica/Malaysian-Qwen2.5-1.5B-Instruct on highly curated Malaysian Reasoning dataset. |
|
|
|
|
|
## Improvement |
|
|
|
|
|
1. Reasoning on Math, Science, Translation, Dialects, Multiple choices, coding and Maktabah Al Bakri. |
|
|
2. Warmup reasoning. |
|
|
|
|
|
## Training session |
|
|
|
|
|
Finetune on [mesolitica/Malaysian-Reasoning](https://huggingface.co/datasets/mesolitica/Malaysian-Reasoning) to make the model better reasoning on Malaysian context. |
|
|
|
|
|
## How we train |
|
|
|
|
|
1. Full parameters on 12k context length. |
|
|
5. WanDB at https://wandb.ai/huseinzol05/fpf-qwen2.5-1.5b-malaysian-12k-reasoning |
|
|
|
|
|
Source code at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5 |
|
|
|
|
|
### Dialect Translation |
|
|
|
|
|
All the benchmarks generate using vLLM, evaluation based on sacrebleu CHRF max@5. |
|
|
|
|
|
Source code for evaluation at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5/evaluate-dialect |
|
|
|
|
|
Dialect to standard Malay, |
|
|
|
|
|
``` |
|
|
|
|
|
``` |
|
|
|
|
|
Standard Malay to dialect, |
|
|
|
|
|
``` |
|
|
|
|
|
``` |
|
|
|
|
|
### MalayMMLU |
|
|
|
|
|
Source code for evaluation at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5/evaluate-malaymmlu |
|
|
|
|
|
Evaluation based on Accuracy@1, |
|
|
|
|
|
``` |
|
|
|
|
|
``` |
|
|
|
|
|
Evaluation based on Accuracy@5, |
|
|
|
|
|
``` |
|
|
|
|
|
``` |
|
|
|
|
|
## Special thanks |
|
|
|
|
|
Special thanks to https://www.sns.com.my and Nvidia for 8x H100 node! |
