PSewmuthu/resnet50-emotion-recognition-ckplus-rafdb

🧠 Emotion Recognition Model – ResNet50 (Fine-Tuned on CK+ and RAF-DB)

📘 Overview

This repository presents a fine-tuned ResNet50-based Emotion Recognition model trained on the CK+ and RAF-DB facial expression datasets. The model classifies facial emotions into seven categories and provides high accuracy and generalization through a two-phase fine-tuning process. Both TensorFlow and TensorFlow Lite versions are available for deployment on cloud and edge devices.

This model forms a key component of an AI-powered, emotion-aware sign language translation system, enabling real-time emotion recognition and integration into affective computing pipelines.

---

🧩 Model Architecture

The architecture is built on ResNet50, pre-trained on ImageNet, and fine-tuned in two stages:

1. Stage 1 – Frozen Base Training (10 Epochs):

   • Convolutional layers frozen.
   • Only top dense layers trained to learn task-specific emotion representations.

2. Stage 2 – Unfrozen Base Fine-Tuning (30 Epochs):

   • Entire base unfrozen with a lower learning rate.
   • Enables deep adaptation to emotion-related facial features.

---

📊 Datasets

1. CK+ Dataset (Kaggle)
2. RAF-DB Dataset (Kaggle)

Dataset Preparation

• Combined CK+ and RAF-DB datasets to increase emotion diversity.
• Applied extensive data augmentation (rotation, flips, brightness, zoom, and shifts).
• Ensured balanced class distribution post-augmentation.

---

⚙️ Training Configuration

Parameter	Description
Base Model	ResNet50 (Pre-trained on ImageNet)
Optimizer	Adam
Learning Rate	1e-4 (unfrozen phase)
Loss Function	Sparse Categorical Crossentropy
Batch Size	32
Epochs	40 (10 + 30)
Image Size	224x224

---

📈 Performance Summary

Metric	Training	Validation	Testing
Accuracy	98.43%	86.05%	81.65%
Loss	0.0610	0.7748	1.1962

Classification Report

Class	Precision	Recall	F1-Score
0	0.74	0.69	0.72
1	0.53	0.49	0.51
2	0.88	0.31	0.46
3	0.93	0.90	0.91
4	0.74	0.87	0.80
5	0.77	0.75	0.76
6	0.82	0.83	0.82

Overall Accuracy: 81.65% Weighted F1-Score: 0.81

---

🖼️ Visualizations

1. Training Accuracy and Loss

• Graph 1: Training and Validation Accuracy vs Epochs

• Graph 2: Training and Validation Loss vs Epochs

2. Dataset Distributions

• Graph 3: Original Dataset Class Distribution

• Graph 4: Balanced (Augmented) Dataset Class Distribution

3. Evaluation Visuals

• Graph 5: Multi-Class ROC Curves (AUC per class)

• Graph 6: Confusion Matrix (Heatmap)

• Graph 7: Sample Test Results (5 predictions per class)

These visualizations demonstrate consistent learning, balanced data, and strong class-wise recognition.

---

🧩 Model Files

File	Description
emotion_resnet50_model.h5	Full TensorFlow model (≈273 MB)
emotion_resnet50_optimized.tflite	Optimized TensorFlow Lite model (≈23 MB) for mobile deployment

---

🧰 Inference Example

import tensorflow as tf
from tensorflow.keras.preprocessing import image
import numpy as np

# Load original model
model_path = 'emotion_resnet50_model.h5'
model = tf.keras.models.load_model(model_path)

# Prepare input
img = image.load_img('test_face.jpg', target_size=(224, 224))
input_data = np.expand_dims(image.img_to_array(img) / 255.0, axis=0)

# Run inference
pred = model.predict(input_data)
classes = ['Angry', 'Disgust', 'Fear', 'Happy', 'Neutral', 'Sad', 'Surprise']
print("Original Model Prediction:", classes[np.argmax(pred)])

# Load and run TFLite optimized model
tflite_model_path = 'emotion_resnet50_optimized.tflite'
interpreter = tf.lite.Interpreter(model_path=tflite_model_path)
interpreter.allocate_tensors()
input_index = interpreter.get_input_details()[0]['index']
output_index = interpreter.get_output_details()[0]['index']
interpreter.set_tensor(input_index, input_data.astype(np.float32))
interpreter.invoke()
output = interpreter.get_tensor(output_index)
print("TFLite Model Prediction:", classes[np.argmax(output)])

---

🚀 Key Features

• Dual-dataset fine-tuning (CK+ + RAF-DB)
• Data-balanced training via augmentation
• High accuracy with stable generalization
• Lightweight TensorFlow Lite version for edge devices
• Ideal for integration in emotion-aware systems

---

🏷️ Tags

emotion-recognition resnet50 facial-expression deep-learning tensorflow tflite ckplus rafdb computer-vision affective-computing fine-tuning

---

📄 Citation

@misc{pasindu_sewmuthu_abewickrama_singhe_2025,
    author       = { Pasindu Sewmuthu Abewickrama Singhe },
    title        = { resnet50-emotion-recognition-ckplus-rafdb (Revision 3216038) },
    year         = 2025,
    url          = { https://huggingface.co/PSewmuthu/resnet50-emotion-recognition-ckplus-rafdb },
    doi          = { 10.57967/hf/6653 },
    publisher    = { Hugging Face }
}

---

👤 Author & Model Info

Author: P.S. Abewickrama Singhe
Developed with: TensorFlow + Keras
License: Apache-2.0
Date: October 2025
Email: [email protected]