Upload folder using huggingface_hub

README.md

@@ -1,6 +1,23 @@
-# GPT-Neo 2.7B Fine-tuned for Lyrics Generation
-
-This model is a fine-tuned version of [EleutherAI/gpt-neo-2.7B](https://huggingface.co/EleutherAI/gpt-neo-2.7B) optimized for generating creative song lyrics based on themes and musical styles.
+---
+language: en
+license: mit
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+  - gpt-neo
+  - causal-lm
+  - text-generation
+  - lora
+  - lyrics
+  - peft
+  - adapter
+datasets:
+  - smgriffin/modern-pop-lyrics
+---
+
+# GPT-Neo 2.7B Fine-tuned LoRA Adapter for Lyrics Generation
+
+This is a **LoRA adapter** for [EleutherAI/gpt-neo-2.7B](https://huggingface.co/EleutherAI/gpt-neo-2.7B) that was fine-tuned to generate creative song lyrics based on themes and musical styles.
 
 ## Model Description
 
@@ -10,22 +27,33 @@ The model has been fine-tuned on a diverse collection of song lyrics to capture
 
 - Base model: GPT-Neo 2.7B
 - Architecture: Transformer-based autoregressive language model
-- Parameters: 2.7 billion
+- Fine-tuning: LoRA (Low-Rank Adaptation) with PEFT
+- Parameters: Full model 2.7 billion, adapter weights much smaller
 - Context window: 2048 tokens
-- Training approach: Fine-tuning on lyrics dataset
+- Training approach: Parameter-efficient fine-tuning on lyrics dataset
 
 ## Usage
 
+This is a LoRA adapter model and must be loaded using the PEFT library:
+
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel, PeftConfig
+
+# Load the base model and tokenizer
+base_model = "EleutherAI/gpt-neo-2.7B"
+adapter_model = "jacob-c/gptneo-2.7Bloratunning"
+
+tokenizer = AutoTokenizer.from_pretrained(base_model)
+base_model = AutoModelForCausalLM.from_pretrained(base_model)
 
-tokenizer = AutoTokenizer.from_pretrained("jacob-c/gptneo-2.7Bloratunning")
-model = AutoModelForCausalLM.from_pretrained("jacob-c/gptneo-2.7Bloratunning")
+# Load the LoRA adapter
+model = PeftModel.from_pretrained(base_model, adapter_model)
 
+# Generate lyrics
 prompt = "Write lyrics for a song with the following themes: love, summer, memories. The lyrics should be:"
 inputs = tokenizer(prompt, return_tensors="pt")
 
-# Generate text
 outputs = model.generate(
     inputs.input_ids,
     max_length=300,
@@ -59,6 +87,16 @@ Nothing's lost and nothing dies
 In this moment frozen in time
 ```
 
+## LoRA Adapter Details
+
+This model uses Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method that significantly reduces the number of trainable parameters by adding pairs of rank-decomposition matrices to existing weights while freezing the original parameters.
+
+LoRA configuration:
+- r: 16
+- alpha: 32
+- Target modules: q_proj, k_proj, v_proj, out_proj
+- Dropout: 0.05
+
 ## Training Process
 
 The model was fine-tuned on lyrics from multiple genres, focusing on: