alvanalrakib/qwen3-4b-reasoning-merge

🎵 Qwen3-4B Lyrics Creation Model

A fine-tuned Qwen3-4B model specialized in step-by-step lyrics creation and songwriting

🤗 Model Card • 💻 GGUF Version • 🔧 Axolotl Framework

---

🌟 Model Overview

This model is a specialized version of Qwen3-4B fine-tuned specifically for lyrics creation and songwriting. It excels at:

• 🎵 Step-by-step lyric writing with visible creative process
• 🎶 Song structure planning (verses, chorus, bridge)
• 💭 Emotional storytelling through lyrics
• 🎤 Various music genres and styles
• ✍️ Creative writing process for songs

📊 Training Performance & Metrics

🏆 Training Results

Metric	Value	Improvement
Initial Loss	2.97	-
Final Eval Loss	1.37	54% reduction
Final Train Loss	1.43	52% reduction
Training Convergence	Excellent	Stable plateau
Overfitting	None detected	Healthy train/eval gap

📈 Loss Progression

• Steps 0-100: Rapid learning phase (2.97 → 1.46)
• Steps 100-400: Pattern mastery (1.46 → 1.37)
• Steps 400-600: Fine convergence (1.37 → 1.35)
• Steps 600-1000: Final optimization (1.35 → 1.37)

⏱️ Training Efficiency

• Total Training Time: 56 minutes 54 seconds
• Steps per Second: 0.291
• Samples per Second: 2.324
• GPU Memory Usage: 26.8GB / 40GB (67% utilization)
• System RAM Usage: 7.1GB / 83.5GB (8% utilization)

📋 Training Configuration

🔧 Model Architecture

Parameter	Value
Base Model	Qwen/Qwen3-4B
Training Method	LoRA Fine-tuning
Total Parameters	4.09B
Trainable Parameters	66.06M (1.62%)
Context Length	4,096 tokens

⚙️ LoRA Configuration

adapter: lora
lora_r: 32
lora_alpha: 64
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj  
  - v_proj
  - o_proj
  - gate_proj
  - down_proj
  - up_proj

🎯 Training Hyperparameters

# Training Schedule
max_steps: 1000
learning_rate: 0.0003
lr_scheduler: cosine
warmup_steps: 100

# Batch Configuration  
micro_batch_size: 4
gradient_accumulation_steps: 2
effective_batch_size: 8

# Optimization
sequence_len: 4096
sample_packing: false
gradient_checkpointing: true

# Precision & Performance
bf16: true
flash_attention: true

📊 Dataset Information

• Dataset Type: High-quality lyrics creation dataset (private)
• Total Samples: 3,500 reasoning examples
• Training Split: 90% (3,150 samples)
• Validation Split: 10% (350 samples)
• Format: Chat template with thinking process
• Specialization: Step-by-step lyrics creation

🔍 Evaluation Metrics

• Eval Steps: Every 100 training steps (10 evaluations total)
• Save Steps: Every 200 training steps (5 checkpoints)
• Logging Steps: Every 10 steps
• Final Evaluation: 10.373 samples/sec, 2.593 steps/sec

🚀 Usage

Basic Setup

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "alvanalrakib/qwen3-4b-reasoning-merge"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

Optimal Generation Parameters

# For lyrics creation (thinking mode)
generation_config = {
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "min_p": 0.0,
    "do_sample": True,
    "max_new_tokens": 2048
}

📁 Model Formats

• 🤗 Transformers: This repository (ready for inference)
• 💻 GGUF Format: alvanalrakib/Qwen3-4B-Reasoning-Lyrics

📋 Technical Specifications

Specification	Details
Parameters	~4.09B total, 66.06M trainable
Architecture	Transformer (Qwen3)
Context Length	32,768 tokens (native)
Training Context	4,096 tokens
Precision	BF16
Inference Memory	~8GB+ VRAM
Thinking Tokens	<think>...</think>
Special Tokens	`<

🔧 Training Infrastructure

💻 Hardware Used

• GPU: NVIDIA A100 40GB
• Memory: 83.5GB System RAM
• Storage: 235.7GB available disk space
• Training Platform: Google Colab Pro+

🛠️ Software Stack

• Framework: Axolotl
• Base Library: Transformers 4.51.0+
• PyTorch: 2.6.0+ with CUDA support
• Precision: Mixed precision (BF16)
• Flash Attention: Enabled

🙏 Acknowledgments

• Qwen Team at Alibaba Cloud for the exceptional Qwen3-4B base model
• Axolotl Community for the outstanding fine-tuning framework
• Hugging Face for the transformers library and model hosting

📄 License

This model is released under the Apache 2.0 License, following the original Qwen3 licensing terms.

---

🎵 Built for songwriters • Optimized for creativity • Made with ❤️

Specialized in lyrics creation with step-by-step creative process