|
|
--- |
|
|
license: mit |
|
|
--- |
|
|
# DUS Forty Layer Merged Model |
|
|
|
|
|
## Overview |
|
|
The DUS Forty Layer Merged Model leverages a unique layer interlocking strategy, combining layers from the Llama-2-13B and Mistral-7B architectures. This approach optimizes computational efficiency while maintaining competitive performance across various natural language processing tasks. |
|
|
|
|
|
## Model Details |
|
|
- **Architecture**: Based on Llama-2-13B and Mistral-7B |
|
|
- **Layer Arrangement**: The `forty` configuration merges layers from both models, interlocking layers 0–20 with layers 12–32. |
|
|
- **Tokenizer**: Mistral-7B tokenizer is used for encoding and decoding. |
|
|
|
|
|
## Training Details |
|
|
- **Base Models**: |
|
|
- Llama-2-13B: [meta-llama/Llama-2-13b-hf](https://huggingface.co/meta-llama/Llama-2-13b-hf) |
|
|
- Mistral-7B: [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) |
