MuzzammilShah/NeuralNetworks-LanguageModels-3

Batch Normalization for Neural Networks: Makemore (Part 3)

In this repository, I implemented Batch Normalization within a neural network framework to enhance training stability and performance, following Andrej Karpathy's approach in the Makemore - Part 3 video.

Overview

This implementation focuses on:

• Normalizing activations and gradients.
• Addressing initialization issues.
• Utilizing Kaiming initialization to prevent saturation of activation functions.

Additionally, visualization graphs were created at the end to analyze the effects of these techniques on the training process and model performance.

Documentation

For a better reading experience and detailed notes, visit my Road to GPT Documentation Site.

Acknowledgments

Notes and implementations inspired by the Makemore - Part 3 video by Andrej Karpathy.

For more of my projects, visit my Portfolio Site.