Model Card for falcon-lora-imdb

Model Description

This is a Falcon model fine-tuned using LoRA (Low-Rank Adaptation) on the IMDB movie reviews dataset for sentiment analysis.

Developed by: keyegon2024
Model type: Causal Language Model with LoRA adapter
Language(s) (NLP): English
License: Apache 2.0
Finetuned from model: tiiuae/falcon-rw-1b

Model Sources

Repository: https://huggingface.co/keyegon2024/falcon-lora-imdb
Base Model: https://huggingface.co/tiiuae/falcon-rw-1b

Uses

Direct Use

This model can be used for:

Sentiment analysis of movie reviews
Text classification tasks (positive/negative sentiment)
Research in sentiment analysis and LoRA fine-tuning techniques

Downstream Use

The model can be adapted for:

General sentiment analysis on other review types
Further fine-tuning on domain-specific sentiment tasks
Integration into applications requiring sentiment classification

Out-of-Scope Use

This model should not be used for generating harmful content
Not suitable for tasks requiring factual accuracy about movies or real-world events
Not designed for multilingual sentiment analysis

Bias, Risks, and Limitations

The model may exhibit biases present in the IMDB dataset
Performance may vary on reviews from different domains (non-movie content)
As a LoRA adapter, it maintains the base model's limitations
May struggle with sarcasm, irony, or complex emotional nuances

Recommendations

Users should be aware that this model is specifically trained for movie review sentiment and may not generalize well to other text types. Consider the dataset's demographic and linguistic biases when applying to new domains.

How to Get Started with the Model

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
from peft import PeftModel, PeftConfig

# Load the model
model_id = "keyegon2024/falcon-lora-imdb"
peft_config = PeftConfig.from_pretrained(model_id)

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    peft_config.base_model_name_or_path,  # tiiuae/falcon-rw-1b
    trust_remote_code=True,
    device_map="auto"
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, model_id)
model.eval()

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    peft_config.base_model_name_or_path, 
    trust_remote_code=True
)
tokenizer.pad_token = tokenizer.eos_token

# Create inference pipeline
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.8,
    top_k=50,
    top_p=0.95
)

# Example usage
prompt = "The movie was absolutely wonderful because"
result = pipe(prompt)
print(result[0]["generated_text"])

Training Details

Training Data

Dataset: IMDB Movie Reviews Dataset (subset)
Training samples: 1,000 reviews (selected from the full training set)
Full dataset size: 50,000 movie reviews (25,000 positive, 25,000 negative)
Source: Hugging Face datasets library (load_dataset("imdb"))

Training Procedure

Preprocessing

Text tokenization using Falcon-RW-1B tokenizer
Maximum sequence length: 128 tokens
Padding: max_length with truncation
Labels set to input_ids for causal language modeling
Pad token set to eos_token

Training Hyperparameters

Training regime: LoRA fine-tuning with 8-bit training
LoRA rank: 8
LoRA alpha: 16
LoRA dropout: 0.1
Target modules: query_key_value (Falcon-specific)
Bias: none
Per device train batch size: 2
Gradient accumulation steps: 4
Number of epochs: 1
Max sequence length: 128
Mixed precision: fp16
Training samples: 1,000 (subset of IMDB train set)

Evaluation

Testing Data

IMDB test set (25,000 reviews)
Standard train/test split from IMDB dataset

Metrics

Accuracy
F1-score
Precision and Recall for positive/negative classes

Results

Formal evaluation metrics are not yet available. The model was trained on a subset of the IMDB dataset for demonstration and learning purposes. For production use, comprehensive evaluation on the full test set would be recommended.

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Hardware Type: Local GPU (consumer-grade)
Hours used: 0.75 hours (45 minutes)
Cloud Provider: N/A (Local training)
Compute Region: N/A (Local training)
Carbon Emitted: Minimal (short training duration on local hardware)

Technical Specifications

Model Architecture and Objective

Base Architecture: Falcon-RW-1B (1 billion parameter transformer-based causal language model)
Adaptation Method: LoRA (Low-Rank Adaptation) with 8-bit training
LoRA Target Modules: query_key_value layers (Falcon-specific attention modules)
Objective: Causal language modeling fine-tuned for sentiment-aware text generation

Compute Infrastructure

Hardware

Training Environment: Local computer
GPU: Consumer-grade GPU with CUDA support
Training Duration: 45 minutes
Memory Optimization: 8-bit training with LoRA for efficient local training

Software

Framework: PyTorch
Libraries:
- transformers
- peft
- datasets
- torch

Citation

BibTeX:

@misc{falcon-lora-imdb-2024,
  author = {keyegon2024},
  title = {Falcon LoRA IMDB Sentiment Analysis Model},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/keyegon2024/falcon-lora-imdb}
}

Model Card Contact

keyegon2024 via Hugging Face

keyegon2024
/

falcon-lora-imdb