README.md

---
license: apache-2.0
datasets:
- Open-Orca/OpenOrca
base_model:
- meta-llama/Llama-2-7b-hf
---
# llama-2 40 layer model

## Model Overview

LlaMa-DUSFT is a custom variant of the LLaMA-2-7B model created using the DUS (Dynamic Update Strategy) methodology. The original LLaMA-2-7B model consists of 32 layers, and this variant introduces a novel approach to optimize performance by reconfiguring and expanding the layer architecture to 40 layers.

### Key Modifications:

1. Layer Splitting:

  - The original 32 layers of LLaMA-2-7B were duplicated.

  - In one variant, the last 12 layers were removed.

  - In another variant, the first 12 layers were removed.

2. Layer Merging:

  - The two resulting 20-layer segments were combined to form a 40-layer model.

### Purpose:

This architectural modification was designed to test whether the DUS approach with an expanded layer count improves performance compared to the standard LLaMA-2 architecture.

## Training Details

### Dataset:

- The model was trained on a subset of the OpenOrca dataset, consisting of 5,000 samples.

### Training Configuration:

- Batch Size: 1

- Epochs: 3

- Optimizer: AdamW

- Learning Rate: 5e-5

- Software: Colab pro

### Preprocessing:

Data preprocessing followed the guidelines for LLaMA-2 models, ensuring tokenization and alignment were consistent with the original architecture.

## Results and Evaluation

### Performance Metrics:

- Due to the experimental nature of this model, specific evaluation metrics are currently limited.

- Initial results indicate improved adaptability in specific downstream tasks from the OpenOrca dataset.

### Observations:

- The DUS layer modification shows potential for enhancing model depth without significant degradation of performance.

- Further evaluation with larger datasets and varied tasks is required to confirm generalizability.