huseinzol05 commited on
Commit
5ce7be1
·
verified ·
1 Parent(s): 0cabd35

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +5 -18
README.md CHANGED
@@ -48,31 +48,18 @@ Standard Malay to dialect,
48
 
49
  ### MalayMMLU
50
 
51
- Accuracy@5,
 
 
52
 
53
  ```
54
 
55
  ```
56
 
57
- While the original model,
58
 
59
  ```
60
- Model Accuracy shot by_letter category
61
- 0 Qwen2.5-1.5B-Instruct 57.306590 0shot True STEM
62
- 1 Qwen2.5-1.5B-Instruct 52.862595 0shot True Language
63
- 2 Qwen2.5-1.5B-Instruct 51.633420 0shot True Social science
64
- 3 Qwen2.5-1.5B-Instruct 52.554569 0shot True Others
65
- 4 Qwen2.5-1.5B-Instruct 57.224118 0shot True Humanities
66
- {'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
67
- Model : Qwen2.5-1.5B-Instruct
68
- Metric : first
69
- Shot : 0shot
70
- average accuracy 53.69842646512204
71
- accuracy for STEM 57.306590257879655
72
- accuracy for Language 52.862595419847324
73
- accuracy for Social science 51.633420063602195
74
- accuracy for Others 52.554569441112974
75
- accuracy for Humanities 57.22411831626849
76
  ```
77
 
78
  ## Special thanks
 
48
 
49
  ### MalayMMLU
50
 
51
+ Source code for evaluation at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5/evaluate-malaymmlu
52
+
53
+ Evaluation based on Accuracy@1,
54
 
55
  ```
56
 
57
  ```
58
 
59
+ Evaluation based on Accuracy@5,
60
 
61
  ```
62
+
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63
  ```
64
 
65
  ## Special thanks