In the previous post, we talked about RNN, and how performing Backpropagation through time (BPTT) on an unrolled RNN with many time steps can lead to the problems of vanishing / exploding gradients, and difficulties in learning long term dependencies. In this post, we’re going to look at a the LSTM (Long Short Term Memory) […]

RNN and Vanishing/Exploding Gradients

In this post, we’re going to be looking at: Recurrent Neural Networks (RNN) Weight updates in an RNN Unrolling an RNN Vanishing/Exploding Gradient Problem Recurrent Neural Networks A Recurrent Neural Network (RNN) is a variant of neural networks, where in each neuron, the outputs cycle back to themselves, hence being recurrent. This means that each […]