November learnings – Recurrent Neural Networks (RNNs)

Week of 11/15/21 – 11/21/21

References

  1. tf.data.Dataset – How to use window function to break a bigger sequence into smaller ones
  2. Load NumPy data – This tutorial provides an example of loading data from NumPy arrays into a tf.data.Dataset
  3. Using tf.data with tf.keras – Building tensorflow input pipelines
  4. Examples
    1. Simple audio recognition: Recognizing keywords
    2. Text generation with an RNN
    3. Generate music with an RNN

Week of 11/15/21 – 11/21/21

  • RNNs
    • keras.sequential: What is it?
    • The Functional API – Keras The Keras Functional API provides more flexibility in creating models compared to keras.sequential. The overall structure for creating models using the functional API is as follows.
      • First create an input node and define the shape of the input. For example, if the input consists of 784 dimensional vectors. Then, we would define the input node as: inputs = keras.Input(shape=(784,)). This is the shape of one sample in the dataset. The batchsize is not to be included.
      • Then, the first layer in the graph is created that takes in this input shape as follows: dense = layers.Dense(64, activation = “relu”), after we have created a layer, then we pass it the input and to give an output, x x = dense(inputs). Note, 64 is the number of units (neurons) in the Dense layer.
      • Then, we simply add more layers as needed. Let’s add another dense layer with 64 units that takes the input (x) from the previous layer and gives (y) as an output: y = layers.Dense(64, activation = “relu”)(x). Lets’ add a final layer as follows: outputs = layers.Dense(10)(x).
      • Finally, we can create a model that aggregates all the layers as follows: model = model.keras.Model(inputs=inputs, outputs=outputs, name = “myFirstModel”)
      • Print the model summary: model.summary( )
      • We can also plot the model and display the inputs and output shapes for each layer in the graph. See keras tutorial for the commands.
      • Training: This is same as the Sequential models. model.fit( )
      • Evaluation: This is same as the Sequential models. model.evaluate ( )
      • Save the model. model.Save ( )
      • The saved model can then be loaded later on as. models.load_model ( )
      • Inference:
Training and evaluation for Models
  • Example of a Functional API for LSTM
  • When to use Functional API?
keras.Sequential( )

References

  1. Text generation with an RNN – refer for general training techniques for RNNs
  2. Recurrent Neural Networks (RNN) with Keras – MUST READ
  3. Keras layers explanation in detail – MUST READ, explains what is behind the black box in Keras LSTM and other layers
  4. tf.keras.layers.LSTM – MUST READ, explains what is behind LSTM cell in tensorflow
  5. tf.keras.layers.Dense – What is behind the dense layer
  6. Training and Evaluation with built in methods in Keras
  7. Writing a training loop from scratch
  8. Masking and padding with Keras
  9. https://ai.googleblog.com/search/label/Speech%20Recognition – History of Speech Recognition
  10. How to Prepare Univariate Time Series Data for Long Short-Term Memory Networks – read, LSTM expects 3 dimensional data (Number of samples/Sequences, Number of time steps in each sample/sequence, Number of elements in a vector at a particular time step (features)..
    1. Also, LSTM works better in the number of time steps in each sequence is between 200-400 elements – more than this will not work properly
  11. How to Reshape Input Data for Long Short-Term Memory Networks in Keras – read, LSTM expects 3 dimensional data
  12. Techniques to Handle Very Long Sequences with LSTMs – read
    1. Use sequence as is
    2. Truncate sequence
    3. Summarize Sequence
    4. Random Sampling
    5. Use Truncated Backpropagation Through Time – post available to read
    6. Use an Encoder-Decoder Architecture
  13. Data Preparation for Variable Length Input Sequences – read
  14. RNN (Recurrent Neural Network) Tutorial: TensorFlow Example – read, example of how to create a simple RNN model in Tensorflow – goes over how to create batches, etc.
  15. Text classification with an RNN – read
  16. Deep Learning LSTM for Sentiment Analysis in Tensorflow with Keras API -an example
  17. TannerGilbert-Tutorials in Github – Sentiment Anslysis
  18. HandsOn Machine Learning – Processing Sequences using RNN and CNN
  19. HandsOn Machine Learning – NLP with RNNs and Attention
  20. Counting Number of Parameters in Feed Forward Deep Neural Network | Keras

Week of 11/01/21 – 11/07/21

  • RNN
RNN architecture: Source [3]
RNN architecture (Ex: for sentiment analysis). Source [3]

References

  1. RNNs Theory
    1. Understanding LSTM Networks (recommended)
    2. Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM) (youtube video)
    3. Lecture 10 | Recurrent Neural Networks (Stanford lecture)
    4. The Unreasonable Effectiveness of Recurrent Neural Networks – Andrej Karpathy
    5. MIT 6.S191 (2020): Recurrent Neural Networks – shows building RNN in Tensorflow mode.