Building a simple recurrent neural network (RNN) in Python from scratch

Author: Goran Trlin

In this code example, we are describing a basic Recurrent Neural Network (RNN) written in Python from scratch. The sample RNN network works with text sequences (sentences) and uses a small vocabulary of only 17 words, encoded as One-hot vectors. At every step, the network reads 3 words and attempts to predict the next (4th) word.

Full network structure:

  • 3 x 17 input neurons
  • 3 x 17 hidden layer neurons
  • 1 x 17 output layer neurons
  • Hidden layer neurons of one step are connected to hidden layer neuorns of the next one

Key notes:

  • Automatic differentiation (autograd package) is used for generating gradients (backpropagation)
  • Basic text input file contains 4 word sentences.
  • Simple regularization added to prevent learning incorrect sentences (dot separation).

The code for this project can be found on GitHub.