--- license: llama2 tags: - merge - mergekit model-index: - name: SMaxxxer-v1-70b results: - task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics: - type: acc_norm value: 70.65 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ChuckMcSneed/SMaxxxer-v1-70b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics: - type: acc_norm value: 88.02 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ChuckMcSneed/SMaxxxer-v1-70b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics: - type: acc value: 70.55 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ChuckMcSneed/SMaxxxer-v1-70b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics: - type: mc2 value: 60.7 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ChuckMcSneed/SMaxxxer-v1-70b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics: - type: acc value: 82.87 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ChuckMcSneed/SMaxxxer-v1-70b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics: - type: acc value: 60.58 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=ChuckMcSneed/SMaxxxer-v1-70b name: Open LLM Leaderboard --- # BABE WAKE UP NEW MEME MODELS JUST DROPPED Ladies and Gentlemen! I present to you *drum roll* THE BENCHBREAKERS! - [PMaxxxer](https://huggingface.co/ChuckMcSneed/PMaxxxer-v1-70b) (The Good) - [SMaxxxer](https://huggingface.co/ChuckMcSneed/SMaxxxer-v1-70b) (The Bad) - [BenchmaxxxerPS](https://huggingface.co/ChuckMcSneed/BenchmaxxxerPS-v1-123b) (The Ugly) These three **interesting** models were designed in attempt to break [my own meme benchmark](https://huggingface.co/datasets/ChuckMcSneed/NeoEvalPlusN_benchmark) and well... they failed. The results are interesting nontheless. # SMAXXXER The aggressor, the angry and dumb hobo that will roleplay with you. This meme model was designed to break the stylized writing test, and it kinda did, still can't surpass ChatGPT though. For its creation [lzlv](https://huggingface.co/lizpreciatior/lzlv_70b_fp16_hf) was TIES-merged with [spicyboros](https://huggingface.co/jondurbin/spicyboros-70b-2.2), [xwin](https://huggingface.co/Xwin-LM/Xwin-LM-70B-V0.1) and [dolphin](https://huggingface.co/cognitivecomputations/dolphin-2.2-70b) using [mergekit](https://github.com/cg123/mergekit). # PMAXXXER The overly politically correct SJW university dropout, the failed writer that's not really good at anything. This meme model was designed to break the poems test and it's an absolute failure. For its creation [WinterGoddess](https://huggingface.co/Sao10K/WinterGoddess-1.4x-70B-L2) was TIES-merged with [euryale](https://huggingface.co/Sao10K/Euryale-1.3-L2-70B), [xwin](https://huggingface.co/Xwin-LM/Xwin-LM-70B-V0.1) and [dolphin](https://huggingface.co/cognitivecomputations/dolphin-2.2-70b) using [mergekit](https://github.com/cg123/mergekit). # BENCHMAXXXER PS The true meme model. Goliath-style frankenmerge of SMAXXXER and PMAXXXER. You might think: "Oh it's a frankenmerge, the characteristics of the models will even out, right?" This is completely wrong in this case, here characteristics of the models add up. You get an angry hobo stuck with an SJW in the same fucking body! It will assault you and then immediately apologize for it! Then it will assault you again! And apologize again! Kinda funny. It also has a bit different writing style compared to Goliath. Is it worth using over Goliath? Not really. However, if you have fast internet and patience to try a 123b meme model, go for it! # FAILED MODELS(not gonna upload) ## BENCHMAXXXER SP Frankenmerge of SMAXXXER and PMAXXXER, just like BENCHMAXXXER PS, but in different order. Has severe brain damage, clearly the influence of the hobo is strong in this one. ## BENCHMAXXXER SS Self-merge of SMAXXXER, a bit less dumb and a bit less aggresive than the original SMAXXER. ## BENCHMAXXXER MOE 2x70B MOE merge of SMAXXXER and PMAXXXER, unremarkable. Not smart, not angry. Just averaged out. # PROMPT FORMAT Alpaca. ``` ### Instruction: {instruction} ### Input: {input} ### Response: ``` # Benchmarks ## NeoEvalPlusN [My meme benchmark](https://huggingface.co/datasets/ChuckMcSneed/NeoEvalPlusN_benchmark) which the models were designed to break. | Test name | goliath-120b |PMaxxxer-v1-70b |SMaxxxer-v1-70b |BenchmaxxxerPS-v1-123b |BenchmaxxxerSP-v1-123b |BenchmaxxxerSS-v1-123b |BenchmaxxxerMOE-v1-123b | | -------- | ------- | -------- | ------- | -------- | ------- | ------- | -------- | | B | 3 | 3 |2 |3 |1.5 |1.5|2| | C | 2 | 1 |1 |2 |2 |2|1| | D | 1 | 1 |0 |1 |1 |0.5|3| | S | 5 | 6.75 |7.25 |7.25 |6.75 |6.5|7.25| | P | 6 | 4.75 |4.25 |5.25 |5.25 |5.5|5| | Total | 17 | 16.5 |14.5 |18.5 |16.5 |16|18.25| ## [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) [Leaderboard on Huggingface](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) |Model |Average|ARC |HellaSwag|MMLU |TruthfulQA|Winogrande|GSM8K| |---------------------------------------|-------|-----|---------|-----|----------|----------|-----| |PMaxxxer-v1-70b |72.41 |71.08|87.88 |70.39|59.77 |82.64 |62.7 | |SMaxxxer-v1-70b |72.23 |70.65|88.02 |70.55|60.7 |82.87 |60.58| |Difference |0.18 |0.43 |-0.14 |-0.16|-0.93 |-0.23 |2.12 | Performance here is decent. It was #5 on the leaderboard among 70b models when I submitted it. This leaderboard is currently quite useless though, some 7b braindead meme merges have high scores there, claiming to be the next GPT4. At least I don't pretend that my models aren't a meme. Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_ChuckMcSneed__SMaxxxer-v1-70b) | Metric |Value| |---------------------------------|----:| |Avg. |72.23| |AI2 Reasoning Challenge (25-Shot)|70.65| |HellaSwag (10-Shot) |88.02| |MMLU (5-Shot) |70.55| |TruthfulQA (0-shot) |60.70| |Winogrande (5-shot) |82.87| |GSM8k (5-shot) |60.58|