Building a simple recurrent neural network (RNN) in Python from scratch

Author: Goran Trlin

In this code example a basic Recurrent Neural Network (RNN) was written in Python from scratch. The provided network works with text sequences (sentences) and uses a small vocabulary of only 17 words, encoded as One-hot vectors. At every step, the network reads 3 words and attempts to predict the next (4th) word. This type of problem is also known as Causal Language Modelling.

Full network structure:

  • 3 x 17 input neurons
  • 3 x 17 hidden layer neurons
  • 1 x 17 output layer neurons
  • Hidden layer neurons of one step are connected to hidden layer neurons of the next one

Key notes:

  • Automatic differentiation (autograd package) was used for generating gradients (backpropagation)
  • Cross-Entropy was used as the loss function
  • Basic text input file contains 4-word sentences.
  • Simple regularization was added to prevent learning incorrect sentences (dot separation).

The full source code of this project can be found on GitHub.