Model Card for LLaMA-3-2-LoRA-EmotionTune

Model Details

• Model Description:
  LLaMA-3-2-LoRA-EmotionTune is a causal language model fine-tuned using Low-Rank Adaptation (LoRA) on a curated emotion dataset. The dataset consists of user-generated text annotated with emotion labels (sadness, joy, love, anger, fear, or surprise). This fine-tuning enables the model to perform efficient emotion classification while preserving the core strengths of the base LLaMA-3.2-1B-Instruct model.

• Developed by:
  Taha Majlesi

• Funded by (optional):
  tahamajs

• Shared by (optional):
  tahamajs

• Model type:
  Causal Language Model with LoRA-based fine-tuning for emotion classification

• Language(s) (NLP):
  English

• License:
  [Choose a license, e.g., Apache-2.0, MIT]

• Finetuned from model (optional):
  LLaMA-3.2-1B-Instruct

• Model Sources (optional):
  Original LLaMA model and publicly available emotion datasets

• Repository:
  https://huggingface.co/your-username/LLaMA-3-2-LoRA-EmotionTune

• Paper (optional):
  For LoRA: LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
  For LLaMA: [Reference paper details if available]

• Demo (optional):
  [Link to interactive demo if available]

---

Uses

• Direct Use:
  Emotion classification and sentiment analysis on short text inputs.

• Downstream Use (optional):
  Can be integrated into affective computing systems, chatbots, content moderation pipelines, or any application requiring real-time sentiment detection.

• Out-of-Scope Use:
  Not recommended for critical decision-making systems (e.g., mental health diagnostics) or for applications in languages other than English without further adaptation.

---

Bias, Risks, and Limitations

• Bias and Risks:
  The model may inherit biases present in the training data, potentially misclassifying nuanced emotions or reflecting cultural biases in emotional expression.

• Limitations:

  • Limited to six predefined emotion categories.
  • Performance may degrade for longer texts or in ambiguous contexts.
  • The model is fine-tuned on a specific emotion dataset and may not generalize well across all domains.

---

Recommendations

Users (both direct and downstream) should be aware of the model’s inherent biases and limitations. We recommend additional validation and fine-tuning before deploying this model in sensitive or high-stakes environments.

---

How to Get Started with the Model

To load and use the model with Hugging Face Transformers and the PEFT library, try the following code snippet:

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/LLaMA-3.2-1B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/LLaMA-3.2-1B-Instruct")

# Load fine-tuned LoRA adapter
model = PeftModel.from_pretrained(base_model, "your-username/LLaMA-3-2-LoRA-EmotionTune")

# Example usage:
input_text = "I feel so happy today!"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Demo: An interactive demo for LLaMA-3-2-LoRA-EmotionTune is available on Hugging Face Spaces at https://huggingface.co/spaces/your-username/demo-name.

---

Training Details

• Training Data:
  A curated emotion dataset consisting of user-generated text annotated with emotion labels (sadness, joy, love, anger, fear, surprise).

• Training Procedure:
  Fine-tuning was performed on the LLaMA-3.2-1B-Instruct model using the LoRA method, which adapts selected attention layers using a low-rank approach. This method leverages a small subset of trainable parameters while keeping the majority of the model frozen.

• Preprocessing (optional):
  Text normalization, tokenization using the Hugging Face tokenizer, and train/validation splitting.

• Training Hyperparameters:

  • LoRA rank (r): 16
  • lora_alpha: 32
  • lora_dropout: 0.1
  • Learning rate: 2e-5
  • Batch size: 32
  • Epochs: Early stopping applied around epoch 10 to prevent overfitting.

• Speeds, Sizes, Times (optional):
  Training conducted on an NVIDIA Tesla T4 GPU for approximately 10–12 hours.

---

Evaluation

• Testing Data:
  A held-out subset of the emotion-annotated dataset (e.g., 100 samples).

• Factors:
  The evaluation focused on the model’s ability to classify emotions within short text outputs.

• Metrics:
  Accuracy and Micro F1 score.

• Results:
  The fine-tuned model achieved an accuracy and Micro F1 score of approximately 31% on short-text generation tasks (5–100 token outputs), outperforming the base and instruction-tuned models.

---

**Technical Specifications **

• Model Architecture and Objective:
  Based on LLaMA-3.2-1B-Instruct, the model is fine-tuned using LoRA to specifically classify text into emotion categories.

• Compute Infrastructure:
  Hugging Face Transformers, PEFT library, and PyTorch.

• Hardware:
  NVIDIA Tesla T4 or equivalent GPU.

• Software:
  Python, PyTorch, Hugging Face Transformers, PEFT 0.14.0.

---

Citation

BibTeX:

@inproceedings{hu2021lora,
  title={LoRA: Low-Rank Adaptation of Large Language Models},
  author={Hu, Edward J and others},
  booktitle={Proceedings of ICLR},
  year={2021}
}

APA:

Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. In Proceedings of ICLR.

---

**Additional Information **

• Model Card Authors:
  Taha Majlesi

• Model Card Contact:
  [email protected]

• Framework Versions:

  • PEFT: 0.14.0
  • Transformers: [version]
  • PyTorch: [version]

---