Upload folder using huggingface_hub

README.md

@@ -0,0 +1,57 @@
+---
+license: apache-2.0
+tags:
+- merge
+- mergekit
+- lazymergekit
+- mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib
+- VAGOsolutions/Llama-3.1-SauerkrautLM-8b-Instruct
+---
+
+# Llama-3.1-8b-instruct_4bitgs64_hqq_calib-Llama-3.1-SauerkrautLM-8b-Instruct-dare-merge
+
+Llama-3.1-8b-instruct_4bitgs64_hqq_calib-Llama-3.1-SauerkrautLM-8b-Instruct-dare-merge is a sophisticated language model resulting from the strategic merging of two powerful models: [mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib](https://huggingface.co/mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib) and [VAGOsolutions/Llama-3.1-SauerkrautLM-8b-Instruct](https://huggingface.co/VAGOsolutions/Llama-3.1-SauerkrautLM-8b-Instruct). This merge was accomplished using [mergekit](https://github.com/cg123/mergekit), a specialized tool that facilitates precise model blending to optimize performance and synergy between the merged architectures.
+
+## 🧩 Merge Configuration
+
+```yaml
+slices:
+  - sources:
+      - model: mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib
+        layer_range: [0, 31]
+      - model: VAGOsolutions/Llama-3.1-SauerkrautLM-8b-Instruct
+        layer_range: [0, 31]
+merge_method: dare
+base_model: mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib
+parameters:
+  t:
+    - filter: self_attn
+      value: [0, 0.5, 0.3, 0.7, 1]
+    - filter: mlp
+      value: [1, 0.5, 0.7, 0.3, 0]
+    - value: 0.5
+dtype: float16
+```
+
+## Model Features
+
+This merged model combines the advanced capabilities of the HQQ quantized version of Llama-3.1-8B-Instruct with the fine-tuned prowess of the SauerkrautLM variant. The result is a versatile model that excels in both generative tasks and nuanced understanding of multilingual contexts, particularly in German and English. The model is designed to handle a variety of text generation tasks, making it suitable for applications ranging from conversational agents to content creation.
+
+## Evaluation Results
+
+The individual models have demonstrated impressive performance across various benchmarks:
+
+- **mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib**:
+  - ARC (25-shot): 60.49
+  - HellaSwag (10-shot): 80.16
+  - MMLU (5-shot): 68.98
+  - Average performance: 69.51
+
+- **VAGOsolutions/Llama-3.1-SauerkrautLM-8b-Instruct**:
+  - Fine-tuned on German-English data, showcasing significant improvements in multilingual capabilities.
+
+The merged model inherits the strengths of both parent models, providing enhanced performance in multilingual contexts while maintaining the efficiency of the HQQ quantization.
+
+## Limitations
+
+While the merged model benefits from the strengths of both parent models, it may also carry over some limitations. For instance, the potential for uncensored content remains a concern, as noted in the SauerkrautLM documentation. Additionally, the model's performance may vary depending on the specific task and the quality of the input data. Users should be aware of these factors when deploying the model in real-world applications.

dare_config.yaml

@@ -0,0 +1,16 @@
+slices:
+  - sources:
+      - model: mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib
+        layer_range: [0, 31]
+      - model: VAGOsolutions/Llama-3.1-SauerkrautLM-8b-Instruct
+        layer_range: [0, 31]
+merge_method: dare
+base_model: mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib
+parameters:
+  t:
+    - filter: self_attn
+      value: [0, 0.5, 0.3, 0.7, 1]
+    - filter: mlp
+      value: [1, 0.5, 0.7, 0.3, 0]
+    - value: 0.5
+dtype: float16