README.md

---
language: en
license: apache-2.0
library_name: tensorflow
tags:
  - tensorflow
  - keras
  - emotion-recognition
  - resnet50
  - ckplus
  - rafdb
  - fine-tuning
  - computer-vision
  - deep-learning
  - facial-expression
  - affective-computing
  - tflite
model-index:
  - name: emotion_resnet50_model
    results:
      - task:
          type: image-classification
        dataset:
          type: dataset
          name: CK+ & RAF-DB
        metrics:
          - name: accuracy
            type: float
            value: 0.8165
    inference: "Supports TensorFlow and TensorFlow Lite inference"
---

# 🧠 Emotion Recognition Model – ResNet50 (Fine-Tuned on CK+ and RAF-DB)

## 📘 Overview

This repository presents a **fine-tuned ResNet50-based Emotion Recognition model** trained on the **CK+** and **RAF-DB** facial expression datasets. The model classifies facial emotions into seven categories and provides high accuracy and generalization through a two-phase fine-tuning process. Both **TensorFlow** and **TensorFlow Lite** versions are available for deployment on cloud and edge devices.

This model forms a key component of an **AI-powered, emotion-aware sign language translation system**, enabling real-time emotion recognition and integration into affective computing pipelines.

---

## 🧩 Model Architecture

The architecture is built on **ResNet50**, pre-trained on ImageNet, and fine-tuned in two stages:

1. **Stage 1 – Frozen Base Training (10 Epochs):**

   - Convolutional layers frozen.
   - Only top dense layers trained to learn task-specific emotion representations.

2. **Stage 2 – Unfrozen Base Fine-Tuning (30 Epochs):**

   - Entire base unfrozen with a lower learning rate.
   - Enables deep adaptation to emotion-related facial features.

---

## 📊 Datasets

1. **[CK+ Dataset (Kaggle)](https://www.kaggle.com/datasets/shareef0612/ckdataset)**
2. **[RAF-DB Dataset (Kaggle)](https://www.kaggle.com/datasets/shuvoalok/raf-db-dataset)**

### Dataset Preparation

- Combined CK+ and RAF-DB datasets to increase emotion diversity.
- Applied **extensive data augmentation** (rotation, flips, brightness, zoom, and shifts).
- Ensured **balanced class distribution** post-augmentation.

---

## ⚙️ Training Configuration

| Parameter         | Description                        |
| ----------------- | ---------------------------------- |
| **Base Model**    | ResNet50 (Pre-trained on ImageNet) |
| **Optimizer**     | Adam                               |
| **Learning Rate** | 1e-4 (unfrozen phase)              |
| **Loss Function** | Sparse Categorical Crossentropy    |
| **Batch Size**    | 32                                 |
| **Epochs**        | 40 (10 + 30)                       |
| **Image Size**    | 224x224                            |

---

## 📈 Performance Summary

| Metric       | Training | Validation | Testing |
| ------------ | -------- | ---------- | ------- |
| **Accuracy** | 98.43%   | 86.05%     | 81.65%  |
| **Loss**     | 0.0610   | 0.7748     | 1.1962  |

### Classification Report

| Class | Precision | Recall | F1-Score |
| ----- | --------- | ------ | -------- |
| 0     | 0.74      | 0.69   | 0.72     |
| 1     | 0.53      | 0.49   | 0.51     |
| 2     | 0.88      | 0.31   | 0.46     |
| 3     | 0.93      | 0.90   | 0.91     |
| 4     | 0.74      | 0.87   | 0.80     |
| 5     | 0.77      | 0.75   | 0.76     |
| 6     | 0.82      | 0.83   | 0.82     |

**Overall Accuracy:** 81.65%
**Weighted F1-Score:** 0.81

---

## 🖼️ Visualizations

### 1. Training Accuracy and Loss

- **Graph 1:** Training and Validation Accuracy vs Epochs
  ![Training Accuracy](images/Training_Accuracy.png)

- **Graph 2:** Training and Validation Loss vs Epochs
  ![Training Loss](images/Training_Loss.png)

### 2. Dataset Distributions

- **Graph 3:** Original Dataset Class Distribution
  ![Original Dataset Distribution](images/Original_Dataset_Distribution.png)

- **Graph 4:** Balanced (Augmented) Dataset Class Distribution
  ![Balanced Dataset Distribution](images/Balanced_Dataset_Distribution.png)

### 3. Evaluation Visuals

- **Graph 5:** Multi-Class ROC Curves (AUC per class)
  ![ROC Curves](images/ROC_Curves.png)

- **Graph 6:** Confusion Matrix (Heatmap)
  ![Confusion Matrix](images/Confusion_Matrix.png)

- **Graph 7:** Sample Test Results (5 predictions per class)
  ![Sample Test Results](images/Sample_Test_Results.png)

These visualizations demonstrate consistent learning, balanced data, and strong class-wise recognition.

---

## 🧩 Model Files

| File                                | Description                                                    |
| ----------------------------------- | -------------------------------------------------------------- |
| `emotion_resnet50_model.h5`         | Full TensorFlow model (≈273 MB)                                |
| `emotion_resnet50_optimized.tflite` | Optimized TensorFlow Lite model (≈23 MB) for mobile deployment |

---

## 🧰 Inference Example

```python
import tensorflow as tf
from tensorflow.keras.preprocessing import image
import numpy as np

# Load original model
model_path = 'emotion_resnet50_model.h5'
model = tf.keras.models.load_model(model_path)

# Prepare input
img = image.load_img('test_face.jpg', target_size=(224, 224))
input_data = np.expand_dims(image.img_to_array(img) / 255.0, axis=0)

# Run inference
pred = model.predict(input_data)
classes = ['Angry', 'Disgust', 'Fear', 'Happy', 'Neutral', 'Sad', 'Surprise']
print("Original Model Prediction:", classes[np.argmax(pred)])

# Load and run TFLite optimized model
tflite_model_path = 'emotion_resnet50_optimized.tflite'
interpreter = tf.lite.Interpreter(model_path=tflite_model_path)
interpreter.allocate_tensors()
input_index = interpreter.get_input_details()[0]['index']
output_index = interpreter.get_output_details()[0]['index']
interpreter.set_tensor(input_index, input_data.astype(np.float32))
interpreter.invoke()
output = interpreter.get_tensor(output_index)
print("TFLite Model Prediction:", classes[np.argmax(output)])
```

---

## 🚀 Key Features

- Dual-dataset fine-tuning (CK+ + RAF-DB)
- Data-balanced training via augmentation
- High accuracy with stable generalization
- Lightweight TensorFlow Lite version for edge devices
- Ideal for integration in emotion-aware systems

---

## 🏷️ Tags

`emotion-recognition` `resnet50` `facial-expression` `deep-learning` `tensorflow` `tflite` `ckplus` `rafdb` `computer-vision` `affective-computing` `fine-tuning`

---

## 📄 Citation

```bibtex
@misc{pasindu_sewmuthu_abewickrama_singhe_2025,
	author       = { Pasindu Sewmuthu Abewickrama Singhe },
	title        = { resnet50-emotion-recognition-ckplus-rafdb (Revision 3216038) },
	year         = 2025,
	url          = { https://huggingface.co/PSewmuthu/resnet50-emotion-recognition-ckplus-rafdb },
	doi          = { 10.57967/hf/6653 },
	publisher    = { Hugging Face }
}
```

---

## 👤 Author & Model Info

**Author:** P.S. Abewickrama Singhe  
**Developed with:** TensorFlow + Keras  
**License:** Apache-2.0  
**Date:** October 2025  
**Email:** [[email protected]](mailto:[email protected])