deep fake with cnn vite and ensembling

This research presents a novel ensemble-based approach for detecting deepfake images using a combination of Convolutional Neural Networks (CNNs) and Vision Transformers (ViT). The system achieves 94.87% accuracy by leveraging three complementary architectures: a 12-layer CNN, a lightweight 6-layer CNN, and a hybrid CNN-ViT model. Our approach demonstrates robust performance in distinguishing between real and manipulated facial images.

README.md

@@ -1,3 +1,127 @@
----
-license: mit
----
+# Ensemble-Based Deep Learning Architecture for Deepfake Detection
+
+## Abstract
+This research presents a novel ensemble-based approach for detecting deepfake images using a combination of Convolutional Neural Networks (CNNs) and Vision Transformers (ViT). The system achieves 94.87% accuracy by leveraging three complementary architectures: a 12-layer CNN, a lightweight 6-layer CNN, and a hybrid CNN-ViT model. Our approach demonstrates robust performance in distinguishing between real and manipulated facial images.
+
+## 1. Introduction
+With the increasing sophistication of deepfake technology, detecting manipulated images has become crucial for maintaining digital media integrity. This work introduces an ensemble method that combines traditional CNN architectures with modern Vision Transformers to create a robust detection system.
+
+## 2. Architecture
+
+### 2.1 Model Components
+The system consists of three distinct models:
+
+1. **Model A (12-layer CNN)**
+   - Three convolutional blocks
+   - Each block: 2 conv layers + BatchNorm + ReLU + pooling
+   - Input size: 50x50 pixels
+   - Dropout rate: 0.3
+
+2. **Model B (6-layer CNN)**
+   - Lightweight architecture
+   - Three simple conv layers with pooling
+   - Input size: 50x50 pixels
+   - Dropout rate: 0.3
+
+3. **Model C (CNN-ViT Hybrid)**
+   - CNN feature extractor
+   - Vision Transformer (base-16 architecture)
+   - Input size: 224x224 pixels
+   - Pretrained ViT backbone
+
+### 2.2 Ensemble Strategy
+The final prediction is determined through majority voting among the three models, enhancing robustness and reducing individual model biases.
+
+## 3. Implementation Details
+
+### 3.1 Dataset
+- Dataset: Hemg/deepfake-and-real-images
+- Split: 80% training, 20% testing
+- Data augmentation: Resize, normalization
+
+### 3.2 Training Parameters
+- Optimizer: Adam
+- Learning rate: 1e-4
+- Batch size: 32
+- Epochs: 10
+- Loss function: Cross-Entropy
+
+## 4. Results
+
+### 4.1 Performance Metrics
+Based on the test set evaluation:
+
+- **Overall Accuracy**: 94.87%
+- **Classification Report**:
+  - Real Images:
+    - Precision: 0.95
+    - Recall: 0.94
+    - F1-score: 0.94
+  - Fake Images:
+    - Precision: 0.94
+    - Recall: 0.95
+    - F1-score: 0.95
+
+### 4.2 Deployment
+The system is deployed as a FastAPI service, providing real-time inference with confidence scores.
+
+### 4.3 Visuals
+
+![Ensemble Architecture](./assets/findings.png)
+*Figure 1: Findings of the proposed ensemble-based deepfake detection system*
+
+#### Performance Visualization
+![Confusion Matrix](./assets/confusion_matrix.png)
+*Figure 2: Confusion matrix showing model performance on test set*
+
+
+#### Loss Vs Epochs
+![Loss Vs Epochs](./assets/Epoc_vs_loss.png)
+*Figure 3: Loss vs Epochs for individual models*
+
+
+
+#### Accuracy Vs Epochs
+![Accuracy Vs Epochs](./assets/accuracy_vs_epochs.png)
+*Figure 4: Accuracy vs Epochs for individual models*
+
+
+## 5. Technical Requirements
+- Python 3.x
+- PyTorch
+- timm
+- FastAPI
+- PIL
+- scikit-learn
+
+## 6. Usage
+
+### 6.1 API Endpoint
+```python
+POST /predict/
+Input: Image file
+Output: {
+    "prediction": "Real/Fake",
+    "confidence": "percentage"
+}
+```
+
+### 6.2 Model Training
+Run cnn-vit file to train models on your custom dataset!
+
+## 7. Conclusions
+The ensemble approach demonstrates superior performance in deepfake detection, with the combination of traditional CNNs and modern Vision Transformers providing robust and reliable results. The system's high accuracy and balanced precision-recall metrics make it suitable for real-world applications. Although for model C it doesn't perform well on Epoch 10 but still overall Result is Good.
+
+## 8. Future Work
+- Integration of attention mechanisms in CNN models
+- Exploration of different ensemble strategies
+- Extension to video deepfake detection
+- Investigation of model compression techniques
+
+## References
+1. Vision Transformer (ViT) - Dosovitskiy et al., 2020
+2. timm library - Ross Wightman
+3. FastAPI - Sebastián Ramírez
+
+## License
+MIT License

app.py

@@ -0,0 +1,51 @@
+# app.py
+from fastapi import FastAPI, File, UploadFile
+from PIL import Image
+import io, torch
+from collections import Counter
+
+from models import ModelA, ModelB, ModelC, transform_small, transform_large
+
+# 1. spin up FastAPI
+app = FastAPI()
+
+# 2. load your saved weights
+device = torch.device('cpu')
+modelA = ModelA(); 
+modelA.load_state_dict(torch.load('modelA.pth', map_location=device,weights_only=True))
+modelA.eval()
+
+modelB = ModelB()
+modelB.load_state_dict(torch.load('modelB.pth', map_location=device,weights_only=True))
+modelB.eval()
+modelC = ModelC()
+modelC.load_state_dict(torch.load('modelC.pth', map_location=device,weights_only=True))
+modelC.eval()
+
+@app.post("/predict/")
+async def predict(file: UploadFile = File(...)):
+    # read image bytes → PIL
+    data = await file.read()
+    img  = Image.open(io.BytesIO(data)).convert('RGB')
+
+    # preprocess
+    t_small = transform_small(img).unsqueeze(0)  # for A & B
+    t_large = transform_large(img).unsqueeze(0)  # for C
+
+    # run inference
+    votes = []
+    with torch.no_grad():
+        for model, inp in [(modelA, t_small), (modelB, t_small), (modelC, t_large)]:
+            out = model(inp)
+            _, pred = out.max(1)
+            votes.append(int(pred.item()))
+
+    # majority vote + confidence
+    vote_count  = Counter(votes)
+    final_label = vote_count.most_common(1)[0][0]
+    confidence  = vote_count[final_label] / len(votes)
+
+    return {
+        "prediction": "Real" if final_label == 1 else "Fake",
+        "confidence": f"{confidence*100:.1f}%"
+    }

ensemble_config.json

@@ -0,0 +1 @@
+{"models": ["modelA.pth", "modelB.pth", "modelC.pth"], "accuracy": 0.9487482596474637}

modelA.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:570d72aef91239c7ac636ddd14fdc00ddb75eee0a8f994f902df271cf2943b37
+size 10603798

modelB.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86b1ff337ca94507916e756f602a8917ad04c719e87827a89e7cdebb65df7cda
+size 9820010

modelC.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff3dcdc52e47b3b3fd8ba4b67366f540a7bf6e0c6d2b046a299655ac8e77c1ad
+size 344739732

models.py

@@ -0,0 +1,153 @@
+import os
+import numpy as np
+from PIL import Image
+from collections import Counter
+import matplotlib.pyplot as plt  # at top of file
+
+
+import torch
+import torch.nn as nn
+import torch.optim as optim
+from torch.utils.data import Dataset, DataLoader
+
+import torchvision.transforms as transforms
+
+
+import timm
+
+
+# -------------------------------
+# Transformations for different model inputs
+# -------------------------------
+# For Model A and Model B, we use small images (50x50)
+transform_small = transforms.Compose([
+    transforms.Resize((50, 50)),
+    transforms.ToTensor(),
+    transforms.Normalize(mean=[0.5]*3, std=[0.5]*3)
+])
+
+# For Model C, we use larger images (224x224)
+transform_large = transforms.Compose([
+    transforms.Resize((224, 224)),
+    transforms.ToTensor(),
+    transforms.Normalize(mean=[0.5]*3, std=[0.5]*3)
+])
+
+
+# --- Model A: CNN-based network for eye and nose regions (12 layers) ---
+class ModelA(nn.Module):
+    def __init__(self, num_classes=2):
+        super(ModelA, self).__init__()
+        # Three convolutional blocks, each with 2 conv layers + BN, ReLU, pooling and dropout
+        self.block1 = nn.Sequential(
+            nn.Conv2d(3, 32, kernel_size=3, padding=1),
+            nn.ReLU(),
+            nn.BatchNorm2d(32),
+            nn.Conv2d(32, 32, kernel_size=3, padding=1),
+            nn.ReLU(),
+            nn.MaxPool2d(kernel_size=2),
+            nn.Dropout(0.3)
+        )
+        self.block2 = nn.Sequential(
+            nn.Conv2d(32, 64, kernel_size=3, padding=1),
+            nn.ReLU(),
+            nn.BatchNorm2d(64),
+            nn.Conv2d(64, 64, kernel_size=3, padding=1),
+            nn.ReLU(),
+            nn.MaxPool2d(kernel_size=2),
+            nn.Dropout(0.3)
+        )
+        self.block3 = nn.Sequential(
+            nn.Conv2d(64, 128, kernel_size=3, padding=1),
+            nn.ReLU(),
+            nn.BatchNorm2d(128),
+            nn.Conv2d(128, 128, kernel_size=3, padding=1),
+            nn.ReLU(),
+            nn.MaxPool2d(kernel_size=2),
+            nn.Dropout(0.3)
+        )
+        # After three blocks, feature map size for 50x50 input: 50 -> 25 -> ~12 -> ~6
+        self.classifier = nn.Sequential(
+            nn.Flatten(),
+            nn.Linear(128 * 6 * 6, 512),
+            nn.ReLU(),
+            nn.Dropout(0.3),
+            nn.Linear(512, num_classes)
+        )
+
+    def forward(self, x):
+        x = self.block1(x)
+        x = self.block2(x)
+        x = self.block3(x)
+        x = self.classifier(x)
+        return x
+
+
+
+# --- Model B: Simpler CNN-based network (6 layers) ---
+class ModelB(nn.Module):
+    def __init__(self, num_classes=2):
+        super(ModelB, self).__init__()
+        # A lighter CNN architecture: three conv layers with pooling and dropout
+        self.features = nn.Sequential(
+            nn.Conv2d(3, 32, kernel_size=3, padding=1),
+            nn.ReLU(),
+            nn.MaxPool2d(2),
+            nn.Dropout(0.3),
+
+            nn.Conv2d(32, 64, kernel_size=3, padding=1),
+            nn.ReLU(),
+            nn.MaxPool2d(2),
+            nn.Dropout(0.3),
+
+            nn.Conv2d(64, 128, kernel_size=3, padding=1),
+            nn.ReLU(),
+            nn.MaxPool2d(2),
+            nn.Dropout(0.3)
+        )
+        self.classifier = nn.Sequential(
+            nn.Flatten(),
+            nn.Linear(128 * 6 * 6, 512),
+            nn.ReLU(),
+            nn.Dropout(0.3),
+            nn.Linear(512, num_classes)
+        )
+
+    def forward(self, x):
+        x = self.features(x)
+        x = self.classifier(x)
+        return x
+    
+
+# --- Model C: CNN + ViT based network for the entire face ---
+class ModelC(nn.Module):
+    def __init__(self, num_classes=2):
+        super(ModelC, self).__init__()
+        # Feature learning (FL) module: a deep CNN.
+        # For demonstration, we use a simpler CNN here.
+        self.cnn_feature_extractor = nn.Sequential(
+            nn.Conv2d(3, 64, kernel_size=3, padding=1),
+            nn.ReLU(),
+            nn.MaxPool2d(2),
+            nn.Conv2d(64, 128, kernel_size=3, padding=1),
+            nn.ReLU(),
+            nn.MaxPool2d(2),
+            nn.Conv2d(128, 256, kernel_size=3, padding=1),
+            nn.ReLU(),
+            nn.MaxPool2d(2)
+        )
+        # Assume feature map size is reduced appropriately (for 224x224, it becomes roughly 28x28)
+        # Now use a vision transformer module from the timm library.
+        # Note: You may need to install timm (pip install timm).
+        self.vit = timm.create_model('vit_base_patch16_224', pretrained=True)
+        # Replace the head of ViT to match our number of classes.
+        in_features = self.vit.head.in_features
+        self.vit.head = nn.Linear(in_features, num_classes)
+
+    def forward(self, x):
+        # Extract lower-level features (optional fusion)
+        features = self.cnn_feature_extractor(x)
+        # For this demonstration, we are feeding the original image to vit.
+        # In a more advanced implementation, you can fuse the CNN features with ViT.
+        out = self.vit(x)
+        return out