|
|
--- |
|
|
base_model: |
|
|
- Sao10K/70B-L3.3-Cirrus-x1 |
|
|
- TheDrummer/Anubis-70B-v1 |
|
|
- meta-llama/Llama-3.3-70B-Instruct |
|
|
- SicariusSicariiStuff/Negative_LLAMA_70B |
|
|
- Sao10K/L3.1-70B-Hanami-x1 |
|
|
- EVA-UNIT-01/EVA-LLaMA-3.33-70B-v0.1 |
|
|
library_name: transformers |
|
|
tags: |
|
|
- mergekit |
|
|
- merge |
|
|
license: llama3.3 |
|
|
--- |
|
|
So I was really happy with V3.3 but I got some more advice on the tokenizers as it tended to run a little hot... this is an experiment, in thought I might be able to get better creativity with higher temps. Early testing is not very promising in terms of fixing it from running hot though. But in terms of an expanded vocab it seems a success! |
|
|
# merge |
|
|
|
|
|
This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit). |
|
|
|
|
|
## Merge Details |
|
|
### Merge Method |
|
|
|
|
|
This model was merged using the [Linear DELLA](https://arxiv.org/abs/2406.11617) merge method using [meta-llama/Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) as a base. |
|
|
|
|
|
### Models Merged |
|
|
|
|
|
The following models were included in the merge: |
|
|
* [Sao10K/70B-L3.3-Cirrus-x1](https://huggingface.co/Sao10K/70B-L3.3-Cirrus-x1) |
|
|
* [TheDrummer/Anubis-70B-v1](https://huggingface.co/TheDrummer/Anubis-70B-v1) |
|
|
* [SicariusSicariiStuff/Negative_LLAMA_70B](https://huggingface.co/SicariusSicariiStuff/Negative_LLAMA_70B) |
|
|
* [Sao10K/L3.1-70B-Hanami-x1](https://huggingface.co/Sao10K/L3.1-70B-Hanami-x1) |
|
|
* [EVA-UNIT-01/EVA-LLaMA-3.33-70B-v0.1](https://huggingface.co/EVA-UNIT-01/EVA-LLaMA-3.33-70B-v0.1) |
|
|
|
|
|
### Configuration |
|
|
|
|
|
The following YAML configuration was used to produce this model: |
|
|
|
|
|
```yaml |
|
|
models: |
|
|
- model: Sao10K/L3.1-70B-Hanami-x1 |
|
|
parameters: |
|
|
weight: 0.20 |
|
|
density: 0.7 |
|
|
- model: Sao10K/70B-L3.3-Cirrus-x1 |
|
|
parameters: |
|
|
weight: 0.20 |
|
|
density: 0.7 |
|
|
- model: SicariusSicariiStuff/Negative_LLAMA_70B |
|
|
parameters: |
|
|
weight: 0.20 |
|
|
density: 0.7 |
|
|
- model: TheDrummer/Anubis-70B-v1 |
|
|
parameters: |
|
|
weight: 0.20 |
|
|
density: 0.7 |
|
|
- model: EVA-UNIT-01/EVA-LLaMA-3.33-70B-v0.1 |
|
|
parameters: |
|
|
weight: 0.20 |
|
|
density: 0.7 |
|
|
merge_method: della_linear |
|
|
base_model: meta-llama/Llama-3.3-70B-Instruct |
|
|
parameters: |
|
|
epsilon: 0.2 |
|
|
lambda: 1.1 |
|
|
dype: float32 |
|
|
out_dtype: bfloat16 |
|
|
tokenizer: |
|
|
source: Sao10K/L3.1-70B-Hanami-x1 |
|
|
|
|
|
``` |
|
|
|
