This is a draft version of the new article on building a basic Recurrent Neural Network in Python from scratch. Full article is coming soon.
In this article, we are describing a basic Recurrent Neural Network written in Python from scratch. The sample RNN network works with text sequences (sentences) and uses a small vocabulary of only 17 words, encoded as One-hot vectors. At every step, the network reads 3 words and attempts to predict the next (4th) word.
Full network structure:
- 3 x 17 input neurons
- 3 x 17 hidden layer neurons
- 1 x 17 output layer neurons
- Hidden layer neurons of one step are connected to hidden layer neuorns of the next one
Key notes:
- Automatic differentiation (autograd package) is used for generating gradients (backpropagation)
- Basic text input file contains 4 word sentences.
- Simple regularization added to prevent learning incorrect sentences (dot separation).
The code for this project can be found on GitHub.
Related subjects
The following subjects are closely related to the methods presented in this article: