Neural Networks and Deep Learning: Basic Definitions
Perceptron: It is one of the simplest artificial neural networks that consists of an input layer and an output layer . The output layer is composed of Linear Threshold Unit (LTU). There is one LTU in every neuron of the output layer. See Figure 1 below.
Multi Layer Perceptron (MLP): As the name suggests, it is an extension of a Perceptron. MLP has one input layer and one output layer, however, it also has an additional layer in between input and output layer called the hidden layer. See Figure 2 below.
Deep Neural Network (DNN): A Neural Network is called Deep Neural Network when there is more than one hidden layer in between the input layer and the output layer.
Convolutional Neural Network (CNN): A Neural Network is called a CNN when the layers between the input and output are the convolutional layers. These are different than the traditional DNNs in many ways. Firstly, they introduce a concept of convolution. Secondly, they take fixed size 2D input as opposed to other DNNs, which take only 1D input. These work well for image recognition tasks. CNNs introduce several other new terms such as stride, receptive field, padding, filters, feature maps, pooling layer. Some of the well known CNNs are LeNet-5, AlexNet (first NN to stack CNN without pooling layer), GoogLeNet (introduced inception module), ResNet (introduced skip connections, residual units), RCNN, Fast RCNN.
Receptive Field: It is a subset of neurons contained in a rectangular region in the previous layer in the stack to which a particular neuron in the current layer is connected to.
Stride: The distance between two adjacent receptive fields.
Padding: Addition of extra zeros around a layer to maintain size consistency in order to allow mathematical operations to be carried between the two adjacent layers.
Filters: It is of the same size as the receptive field. It contains the weights and convolves with the previous layer to produce an output. The depth of the filter must be the same as the number of channels in the input. The CNNs learn the filters during the training phase.
Feature map: When the same filter is used across the entire layer, the resulting output after the convolution is called a feature map. The feature map highlights the areas in the previous layer that are most similar to the filter. Note: It is possible to apply different types of filters at the same time, resulting in more than one feature map being generated in one layer. The feature maps are also called the activation maps.
Pooling layer: The pooling layer aims to reduce the dimension. It uses techniques such as max-pooling or average pooling to reduce the size of the next layer in the stack.
Recurrent Neural Network (RNN):
Autoencoder:
Figure 1: A Perceptron (This is adopted from reference 2 mentioned below)
Figure 2: A Multi Layer Perceptron (MLP) – This is adopted from reference 3 below
A representation of a typical CNN stack – This is adopted from reference 7 below.
References
Géron, Aurélien. Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems. O’Reilly Media, 2019.
