In this code example a basic Recurrent Neural Network (RNN) was written in Python from scratch. The provided network works with text sequences (sentences) and uses a small vocabulary of only 17 words, encoded as One-hot vectors. At every step, the network reads 3 words and attempts to predict the next (4th) word. This type of problem is also known as Causal Language Modelling.
Full network structure:
- 3 x 17 input neurons
- 3 x 17 hidden layer neurons
- 1 x 17 output layer neurons
- Hidden layer neurons of one step are connected to hidden layer neurons of the next one
- Automatic differentiation (autograd package) was used for generating gradients (backpropagation)
- Cross-Entropy was used as the loss function
- Basic text input file contains 4-word sentences.
- Simple regularization was added to prevent learning incorrect sentences (dot separation).
The full source code of this project can be found on GitHub.