In this code example, we are describing a basic Recurrent Neural Network (RNN) written in Python from scratch. The sample RNN network works with text sequences (sentences) and uses a small vocabulary of only 17 words, encoded as One-hot vectors. At every step, the network reads 3 words and attempts to predict the next (4th) word.
Full network structure:
- 3 x 17 input neurons
- 3 x 17 hidden layer neurons
- 1 x 17 output layer neurons
- Hidden layer neurons of one step are connected to hidden layer neuorns of the next one
- Automatic differentiation (autograd package) is used for generating gradients (backpropagation)
- Basic text input file contains 4 word sentences.
- Simple regularization added to prevent learning incorrect sentences (dot separation).
The code for this project can be found on GitHub.