YAML Metadata
Warning:
empty or missing yaml metadata in repo card
(https://huggingface.co/docs/hub/model-cards#model-card-metadata)
Quantization made by Richard Erkhov.
Gutensuppe-mistral-nemo-12B - GGUF
- Model creator: https://huggingface.co/nbeerbower/
- Original model: https://huggingface.co/nbeerbower/Gutensuppe-mistral-nemo-12B/
Original model description:
library_name: transformers tags: - mergekit - merge base_model: - nbeerbower/mistral-nemo-gutenberg-12B-v3 - nbeerbower/mistral-nemo-gutenberg-12B-v4 - nbeerbower/Lyra-Gutenberg-mistral-nemo-12B - nbeerbower/mistral-nemo-gutenberg-12B model-index: - name: Gutensuppe-mistral-nemo-12B results: - task: type: text-generation name: Text Generation dataset: name: IFEval (0-Shot) type: HuggingFaceH4/ifeval args: num_few_shot: 0 metrics: - type: inst_level_strict_acc and prompt_level_strict_acc value: 29.16 name: strict accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nbeerbower/Gutensuppe-mistral-nemo-12B name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: BBH (3-Shot) type: BBH args: num_few_shot: 3 metrics: - type: acc_norm value: 35.57 name: normalized accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nbeerbower/Gutensuppe-mistral-nemo-12B name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MATH Lvl 5 (4-Shot) type: hendrycks/competition_math args: num_few_shot: 4 metrics: - type: exact_match value: 11.03 name: exact match source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nbeerbower/Gutensuppe-mistral-nemo-12B name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: GPQA (0-shot) type: Idavidrein/gpqa args: num_few_shot: 0 metrics: - type: acc_norm value: 11.63 name: acc_norm source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nbeerbower/Gutensuppe-mistral-nemo-12B name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MuSR (0-shot) type: TAUR-Lab/MuSR args: num_few_shot: 0 metrics: - type: acc_norm value: 14.33 name: acc_norm source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nbeerbower/Gutensuppe-mistral-nemo-12B name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MMLU-PRO (5-shot) type: TIGER-Lab/MMLU-Pro config: main split: test args: num_few_shot: 5 metrics: - type: acc value: 29.78 name: accuracy source: url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=nbeerbower/Gutensuppe-mistral-nemo-12B name: Open LLM Leaderboard
Gutensuppe-mistral-nemo-12B
This is a merge of pre-trained language models created using mergekit.
Merge Details
Merge Method
This model was merged using the Model Stock merge method using nbeerbower/mistral-nemo-gutenberg-12B as a base.
Models Merged
The following models were included in the merge:
- nbeerbower/mistral-nemo-gutenberg-12B-v3
- nbeerbower/mistral-nemo-gutenberg-12B-v4
- nbeerbower/Lyra-Gutenberg-mistral-nemo-12B
Configuration
The following YAML configuration was used to produce this model:
models:
- model: nbeerbower/Lyra-Gutenberg-mistral-nemo-12B
- model: nbeerbower/mistral-nemo-gutenberg-12B-v4
- model: nbeerbower/mistral-nemo-gutenberg-12B-v3
- model: nbeerbower/mistral-nemo-gutenberg-12B
merge_method: model_stock
base_model: nbeerbower/mistral-nemo-gutenberg-12B
dtype: bfloat16
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 21.92 |
| IFEval (0-Shot) | 29.16 |
| BBH (3-Shot) | 35.57 |
| MATH Lvl 5 (4-Shot) | 11.03 |
| GPQA (0-shot) | 11.63 |
| MuSR (0-shot) | 14.33 |
| MMLU-PRO (5-shot) | 29.78 |
- Downloads last month
- 114