Just to obtain metrics from the HuggingFaceH4/open_llm_leaderboard.
To evaluate the impact of increasing the number of experts, modify the num_experts_per_tok setting in the config.json file from 2 to 3. This alteration aims to specifically determine if such a change leads to any notable improvements in performance metrics.
Other details to note include that the model weights are directly copied from the source available at https://huggingface.co/mistralai/Mixtral-8x7B-v0.1.
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 69.09 |
| AI2 Reasoning Challenge (25-Shot) | 67.41 |
| HellaSwag (10-Shot) | 86.63 |
| MMLU (5-Shot) | 71.98 |
| TruthfulQA (0-shot) | 48.58 |
| Winogrande (5-shot) | 82.40 |
| GSM8k (5-shot) | 57.54 |
- Downloads last month
- 1,077
Inference Providers
NEW
This model is not currently available via any of the supported third-party Inference Providers, and
the model is not deployed on the HF Inference API.
Model tree for itsliupeng/Mixtral-8x7B-v0.1-top3
Evaluation results
- normalized accuracy on AI2 Reasoning Challenge (25-Shot)test set Open LLM Leaderboard67.410
- normalized accuracy on HellaSwag (10-Shot)validation set Open LLM Leaderboard86.630
- accuracy on MMLU (5-Shot)test set Open LLM Leaderboard71.980
- mc2 on TruthfulQA (0-shot)validation set Open LLM Leaderboard48.580
- accuracy on Winogrande (5-shot)validation set Open LLM Leaderboard82.400
- accuracy on GSM8k (5-shot)test set Open LLM Leaderboard57.540