---
tags:
- merge
- mergekit
- lazymergekit
- flemmingmiguel/NeuDist-Ro-7B
- johannhartmann/Brezn3
- ResplendentAI/Flora_DPO_7B
base_model:
- flemmingmiguel/NeuDist-Ro-7B
- johannhartmann/Brezn3
- ResplendentAI/Flora_DPO_7B
language:
- de
- en
---

# Spaetzle-v8-7b

This model is supposed to show adequate performance in German and English on a number of tasks, while mostly behaving well, that is, without rambling on, intermixing tokens from different templates in training and adapting, etc.

It is mostly a quick test, and considerably weaker in German grammar and orthography than DiscoLM e.g., but for use cases where this is not too important, but e.g. instruction following, reasoning, etc, it might actually be a little bit preferable.

It is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
* [flemmingmiguel/NeuDist-Ro-7B](https://huggingface.co/flemmingmiguel/NeuDist-Ro-7B)
* [johannhartmann/Brezn3](https://huggingface.co/johannhartmann/Brezn3)
* [ResplendentAI/Flora_DPO_7B](https://huggingface.co/ResplendentAI/Flora_DPO_7B)
* on the basis of [mayflowergmbh/Wiedervereinigung-7b-dpo-laser](https://huggingface.co/mayflowergmbh/Wiedervereinigung-7b-dpo-laser)

All credits are due to the creators of those original models and the training datasets involved.

For a suitable quantized version, try [cstr/Spaetzle-v8-7b-GGUF](https://huggingface.co/cstr/Spaetzle-v8-7b-GGUF)


## Evaluation
[Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_cstr__Spaetzle-v8-7b)

|             Metric              |Value|
|---------------------------------|----:|
|Avg.                             |72.27|
|AI2 Reasoning Challenge (25-Shot)|68.69|
|HellaSwag (10-Shot)              |86.68|
|MMLU (5-Shot)                    |64.60|
|TruthfulQA (0-shot)              |64.05|
|Winogrande (5-shot)              |81.45|
|GSM8k (5-shot)                   |68.16|

EQ-Bench (v2_de): 61.04 / english (v2): 78.3

[ScandEval](https://scandeval.com/german-nlg/) 12.5.2 scores 

| Benchmark             | Spaetzle-v8-7b Value                               |
|-----------------------|----------------------------------------------------|
| Model ID              | cstr/Spaetzle-v8-7b (few-shot, val)                |
| Parameters            | 7242                                               |
| Vocabulary Size       | 32                                                 |
| Context               | 32768                                              |
| Commercial            | False                                              |
| Speed                 | 5,980 ± 1,031 / 1,714 ± 552                        |
| Rank                  | 1.85                                               |
| GermEval              | 58.90 ± 2.30 / 45.55 ± 3.30                        |
| SB10k                 | 61.34 ± 1.90 / 72.98 ± 1.30                        |
| ScaLA-De              | 31.58 ± 4.39 / 65.51 ± 2.23                        |
| GermanQuAD            | 24.91 ± 3.98 / 60.88 ± 3.31                        |
| MLSum                 | 67.25 ± 1.06 / 22.95 ± 2.64                        |
| MMLU-De               | 34.62 ± 2.20 / 50.43 ± 1.52                        |
| HellaSwag-De          | 48.70 ± 2.47 / 61.05 ± 1.79                        |


|                           Model                            |AGIEval|GPT4All|TruthfulQA|Bigbench|Average|
|------------------------------------------------------------|------:|------:|---------:|-------:|------:|
|[Spaetzle-v8-7b](https://huggingface.co/cstr/Spaetzle-v8-7b)|  45.31|  75.69|     63.94|   45.57|  57.63|

### AGIEval
|             Task             |Version| Metric |Value|   |Stderr|
|------------------------------|------:|--------|----:|---|-----:|
|agieval_aqua_rat              |      0|acc     |25.59|±  |  2.74|
|                              |       |acc_norm|24.80|±  |  2.72|
|agieval_logiqa_en             |      0|acc     |39.63|±  |  1.92|
|                              |       |acc_norm|39.78|±  |  1.92|
|agieval_lsat_ar               |      0|acc     |23.48|±  |  2.80|
|                              |       |acc_norm|24.35|±  |  2.84|
|agieval_lsat_lr               |      0|acc     |50.98|±  |  2.22|
|                              |       |acc_norm|51.96|±  |  2.21|
|agieval_lsat_rc               |      0|acc     |62.08|±  |  2.96|
|                              |       |acc_norm|62.83|±  |  2.95|
|agieval_sat_en                |      0|acc     |78.64|±  |  2.86|
|                              |       |acc_norm|79.13|±  |  2.84|
|agieval_sat_en_without_passage|      0|acc     |44.66|±  |  3.47|
|                              |       |acc_norm|44.66|±  |  3.47|
|agieval_sat_math              |      0|acc     |37.27|±  |  3.27|
|                              |       |acc_norm|35.00|±  |  3.22|

Average: 45.31%

### GPT4All
|    Task     |Version| Metric |Value|   |Stderr|
|-------------|------:|--------|----:|---|-----:|
|arc_challenge|      0|acc     |63.14|±  |  1.41|
|             |       |acc_norm|64.51|±  |  1.40|
|arc_easy     |      0|acc     |85.98|±  |  0.71|
|             |       |acc_norm|82.49|±  |  0.78|
|boolq        |      1|acc     |88.10|±  |  0.57|
|hellaswag    |      0|acc     |66.31|±  |  0.47|
|             |       |acc_norm|85.17|±  |  0.35|
|openbookqa   |      0|acc     |38.00|±  |  2.17|
|             |       |acc_norm|47.20|±  |  2.23|
|piqa         |      0|acc     |83.35|±  |  0.87|
|             |       |acc_norm|84.17|±  |  0.85|
|winogrande   |      0|acc     |78.22|±  |  1.16|

Average: 75.69%

### TruthfulQA
|    Task     |Version|Metric|Value|   |Stderr|
|-------------|------:|------|----:|---|-----:|
|truthfulqa_mc|      1|mc1   |47.74|±  |  1.75|
|             |       |mc2   |63.94|±  |  1.53|

Average: 63.94%

### Bigbench
|                      Task                      |Version|       Metric        |Value|   |Stderr|
|------------------------------------------------|------:|---------------------|----:|---|-----:|
|bigbench_causal_judgement                       |      0|multiple_choice_grade|56.84|±  |  3.60|
|bigbench_date_understanding                     |      0|multiple_choice_grade|66.12|±  |  2.47|
|bigbench_disambiguation_qa                      |      0|multiple_choice_grade|41.47|±  |  3.07|
|bigbench_geometric_shapes                       |      0|multiple_choice_grade|22.01|±  |  2.19|
|                                                |       |exact_str_match      | 0.00|±  |  0.00|
|bigbench_logical_deduction_five_objects         |      0|multiple_choice_grade|31.40|±  |  2.08|
|bigbench_logical_deduction_seven_objects        |      0|multiple_choice_grade|23.14|±  |  1.60|
|bigbench_logical_deduction_three_objects        |      0|multiple_choice_grade|56.00|±  |  2.87|
|bigbench_movie_recommendation                   |      0|multiple_choice_grade|45.00|±  |  2.23|
|bigbench_navigate                               |      0|multiple_choice_grade|50.70|±  |  1.58|
|bigbench_reasoning_about_colored_objects        |      0|multiple_choice_grade|70.05|±  |  1.02|
|bigbench_ruin_names                             |      0|multiple_choice_grade|45.54|±  |  2.36|
|bigbench_salient_translation_error_detection    |      0|multiple_choice_grade|26.05|±  |  1.39|
|bigbench_snarks                                 |      0|multiple_choice_grade|71.82|±  |  3.35|
|bigbench_sports_understanding                   |      0|multiple_choice_grade|72.92|±  |  1.42|
|bigbench_temporal_sequences                     |      0|multiple_choice_grade|44.20|±  |  1.57|
|bigbench_tracking_shuffled_objects_five_objects |      0|multiple_choice_grade|22.80|±  |  1.19|
|bigbench_tracking_shuffled_objects_seven_objects|      0|multiple_choice_grade|18.23|±  |  0.92|
|bigbench_tracking_shuffled_objects_three_objects|      0|multiple_choice_grade|56.00|±  |  2.87|

Average: 45.57%

Average score: 57.63%

## 💻 Usage

```python
!pip install -qU transformers accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "cstr/Spaetzle-v8-7b"
messages = [{"role": "user", "content": "What is a large language model?"}]

tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
```


## 🧩 Configuration

The model uses ChatML and should work well with this (as it is merged from models which (mostly) saw ChatML templates in training). 

```yaml
models:
  - model: mayflowergmbh/Wiedervereinigung-7b-dpo-laser
    # no parameters necessary for base model
  - model: flemmingmiguel/NeuDist-Ro-7B
    parameters:
      density: 0.60
      weight: 0.30
  - model: johannhartmann/Brezn3
    parameters:
      density: 0.65
      weight: 0.40
  - model: ResplendentAI/Flora_DPO_7B
    parameters:
      density: 0.6
      weight: 0.3
merge_method: dare_ties
base_model: mayflowergmbh/Wiedervereinigung-7b-dpo-laser
parameters:
  int8_mask: true
dtype: bfloat16
random_seed: 0
tokenizer_source: base
```