README.md

metadata

language: en
library_name: transformers
license: llama3.1
base_model: meta-llama/Llama-3.1-8B-Instruct
pipeline_tag: text-generation
tags:
  - llama-3.1
  - instruction-tuned
datasets:
  - OpenAssistant/oasst1
  - databricks/databricks-dolly-15k
  - Open-Orca/OpenOrca
  - mlabonne/open-perfectblend
  - tatsu-lab/alpaca
model-index:
  - name: utkmst/chimera-beta-test2-lora-merged
    results:
      - task:
          type: text-generation
        dataset:
          type: leaderboard
          name: Overall Leaderboard
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.444
            verified: true
          - name: acc
            type: acc
            value: 0.2992
            verified: true
          - name: exact_match
            type: exact_match
            value: 0.0951
            verified: true
      - task:
          type: text-generation
        dataset:
          type: bbh
          name: BBH (Big Bench Hard)
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.4773
            verified: true
      - task:
          type: text-generation
        dataset:
          type: gpqa
          name: GPQA (Google-Patched Question Answering)
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.3036
            verified: true
      - task:
          type: text-generation
        dataset:
          type: math
          name: Math
        metrics:
          - name: exact_match
            type: exact_match
            value: 0.0951
            verified: true
      - task:
          type: text-generation
        dataset:
          type: mmlu_pro
          name: MMLU-Pro
        metrics:
          - name: acc
            type: acc
            value: 0.2992
            verified: true
      - task:
          type: text-generation
        dataset:
          type: musr
          name: MUSR (Multi-Step Reasoning)
        metrics:
          - name: acc_norm
            type: acc_norm
            value: 0.4113
            verified: true

utkmst/chimera-beta-test2-lora-merged

Model Description

This model is a fine-tuned version of Meta's Llama-3.1-8B-Instruct model, created through LoRA fine-tuning on multiple instruction datasets, followed by merging the adapter weights with the base model.

Architecture

• Base Model: meta-llama/Llama-3.1-8B-Instruct
• Size: 8.03B parameters
• Type: Decoder-only transformer
• Format: SafeTensors (full precision)

Training Details

• Training Method: LoRA fine-tuning followed by adapter merging
• LoRA Configuration:
  • Rank: 8
  • Alpha: 16
  • Trainable modules: Attention layers and feed-forward networks
• Training Hyperparameters:
  • Learning rate: 2e-4
  • Batch size: 2
  • Training epochs: 1
  • Optimizer: AdamW with constant scheduler

Intended Use

This model is designed for:

• General purpose assistant capabilities
• Question answering and knowledge retrieval
• Creative content generation
• Instructional guidance

Limitations

• Base model limitations including potential hallucinations and factual inaccuracies
• Limited context window compared to larger models
• Knowledge cutoff from the base Llama-3.1 model
• May exhibit biases present in training data
• Performance on specialized tasks may vary

Usage with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("utkmst/chimera-beta-test2-lora-merged")
tokenizer = AutoTokenizer.from_pretrained("utkmst/chimera-beta-test2-lora-merged")

License

This model inherits the license from Meta's Llama 3.1.