MuzzammilShah/NeuralNetworks-LanguageModels-4

Manual Backpropagation through BatchNorm: Makemore (Part 4)

This repository explores manual backpropagation through a 2-layer MLP (with BatchNorm) without using PyTorch autograd's loss.backward(). It involves manually backpropagating through the cross entropy loss, 2nd linear layer, tanh, batchnorm, 1st linear layer, and the embedding table.

Documentation

For a better reading experience and detailed notes, visit my Road to GPT Documentation Site.

Acknowledgments

Notes and implementations inspired by the Makemore - Part 4 video by Andrej Karpathy.

For more of my projects, visit my Portfolio Site.