The previous article was about the process of convolution and its implementation. This article is about the padding, stride and the parameters involved in a CNN.

We have seen that there is a reduction of dimension in the output vector. A technique known as padding is done to preserve the original dimensions in the output vector. The only change in this process is that we add a boundary of ‘0s’ over the input vector and then do the convolution process.

Procedure to implement padding

1. To get n*n output use a (n+2*n+2) input
2. To get 7*7 output use 9*9 input
3. In that 9*9 input fill the first row, first column, last row and last column with zero.
4. Now do the convolution operation on it using a filter.
5. Observe that the output has the same dimensions as of the input.

Zero is used since it is insignificant so as to keep the output dimension without affecting the results

Here all the elements in the input vector have been transferred to the output. Hence using padding we can preserve the originality of the input. Padding is denoted using P. If P=1 then one layer of zeroes is added and so on.

It is not necessary that the filter or kernel must be applied to all the cells. The pattern of applying the kernel onto the input vector is determined using the stride. It determines the shift or gaps in the cells where the filter has to be applied.-

S=1 means no gap is created. The filter is applied to all the cells.

S=2 means gap of 1. The filter is applied to alternative cells. This halves the dimensions on the output vector.

This diagram shows the movement of filter on a vector with stride of 1 and 2. With a stride of 2; alternative columns are accessed and hence the number of computations per row decreases by 2. Hence the output dimensions reduce while use stride.

The padding and stride are some features used in CNN.

Parameters in a convolution layer

The following are the terms needed for calculating the parameter for a convolution layer.

Input layer

Width Wi – width of input image

Height Hi – height of input image

Depth Di – 3 since they follow RGB

We saw that 7*7 inputs without padding and stride along with 3*3 kernels gave a 5*5 output. It can be verified using this calculation.

The role of padding can also be verified using this calculation.

The f is known as filter size. It can be a 1*1, 3*3 and so on. It is a 1-D value so the first value is taken. There is another term K which refers to the number of kernels used. This value is fixed by user.

These values are similar to those of w and b. The machine learns the ideal value for these parameters for high efficiency. The significance of partial connection or CNN can be easily understood through the parameters.

Consider the same example of (30*30*3) vector. The parameter for CNN by using 10 kernels will be 2.7 million. This is a large number. But if the same is done using FNN then the parameters will be at least 100 million. This is almost 50 times that of before. This is significantly larger than CNN. The reason for this large number is due to the full connectivity. 

                                                 

Parameter= 30*30*3*3*10= 2.7M

HAPPY READING!!

The previous article was about the procedure to develop a deep learning network and introduction to CNN. This article concentrates on the process of convolution which is the process of taking in two images and doing a transformation to produce an output image. This process is common in mathematics and signals analysis also. The CNN’s are mainly used to work with images.

In the CNN partial connection is observed. Hence all the neurons are not connected to those in the next layer. So the number of parameters reduces leading to lesser computations.

Sample connection is seen in CNN.

Convolution in mathematics refers to the process of combining two different functions. With respect to CNN, convolution occurs between the image and the filter or kernel. Convolution itself is one of the processes done on the image.

Here also the operation is mathematical. It is a kind of operation on two vectors. The input image gets converted into a vector based on colour and dimension. The kernel or filter is a predefined vector with fixed values to perform various functions onto the image.

Process of convolution

The kernel or filter is chosen in order of 1*1, 3*3, 5*5, 7*7, and so on. The given filter vector slides over the image and performs dot product over the image vector and produces an output vector with the result of each 3*3 dot product over the 7*7 vector.

A 3*3 kernel slides over the 7*7 input vector to produce a 5*5 output image vector. The reason for the reduction in the dimension is that the kernel has to do dot product operation on the input vector-only with the same dimension. I.e. the kernel slides for every three rows in the seven rows. The kernel must perfectly fit into the input vector. All the cells in the kernel must superimpose onto the vector. No cells must be left open. There are only 5 ways to keep a 3-row filter in a 7-row vector.    

This pictorial representation can help to understand even better. These colors might seem confusing, but follow these steps to analyze them.

1. View at the first row.
2. Analyse and number the different colours used in that row
3. Each colour represents a 3*3 kernel.
4. In the first row the different colours are red, orange, light green, dark green and blue.
5. They count up to five.
6. Hence there are five ways to keep a 3 row filter over a 7 row vector.
7. Repeat this analysis for all rows
8. 35 different colours will be used. The math is that in each row there will be 5 combinations. For 7 rows there will be 35 combinations.
9. The colour does not go beyond the 7 rows signifying that kernel cannot go beyond the dimension of input vector.

These are the 35 different ways to keep a 3*3 filter over a 7*7 image vector. From this diagram, we can analyse each row has five different colours. All the nine cells in the kernel must fit inside the vector. This is the reason for the reduction in the dimension of output vector.

Procedure to implement convolution

1. Take the input image with given dimensions.
2. Flatten it into 1-D vector. This is the input vector whose values represent the colour of a pixel in the image.
3. Decide the dimension, quantity and values for filter. The value in a filter is based on the function needed like blurring, fadening, sharpening etc. the quantity and dimension is determined by the user.
4. Take the filter and keep it over the input vector from the first cell. Assume a 3*3 filter kept over a 7*7 vector.
5. Perform the following computations on them.

5a. take the values in the first cell of the filter and the vector.

5b. multiply them.

5c. take the values in the second cell of the filter and the vector.

5d. multiply them.

5e. repeat the procedure till the last cell.

5f. take the sum for all the nine values.

• Place this value in the output vector.
• Using the formula mentioned later, find the dimensions of the output vector.

HAPPY LEARNING!!

The previous article dealt with the networks and the backpropagation algorithm. This article is about the mathematical implementation of the algorithm in FFN followed by an important concept called hyper-parameter tuning.

In this FFN we apply the backpropagation to find the partial derivative of the loss function with respect to w1 so as to update w1.

Hence using backpropagation the algorithm determines the update required in the parameters so as to match the predicted output with the true output. The algorithm which performs this is known as Vanilla Gradient Descent.

The way of reading the input is determined using the strategy.

Strategy	Meaning
Stochastic	One by one
Batch	Splitting entire input into batches
Mini-batch	Splitting batch into batches

The sigmoid here is one of the types of the activation function. It is defined as the function pertaining to the transformation of input to output in a particular neuron. Differentiating the activation function gives the respective terms in the gradients.

There are two common phenomena seen in training networks. They are

1. Under fitting
2. Over fitting

If the model is too simple to learn the data then the model can underfit the data. In that case, complex models and algorithms must be used.

If the model is too complex to learn the data then the model can overfit the data. This can be visualized by seeing the differences in the training and testing loss function curves. The method adopted to change this is known as regularisation. Overfit and underfit can be visualized by plotting the graph of testing and training accuracies over the iterations. Perfect fit represents the overlapping of both curves.

Regularisation is the procedure to prevent the overfitting of data. Indirectly, it helps in increasing the accuracy of the model. It is either done by

1. Adding noises to input to affect and reduce the output.
2. To find the optimum iterations by early stopping
3. By normalising the data (applying normal distribution to input)
4. By forming subsets of a network and training them using dropout.

So far we have seen a lot of examples for a lot of procedures. There will be confusion arising at this point on what combination of items to use in the network for maximum optimization. There is a process known as hyper-parameter tuning. With the help of this, we can find the combination of items for maximum efficiency. The following items can be selected using this method.

1. Network architecture
2. Number of layers
3. Number of neurons in each layer
4. Learning algorithm
5. Vanilla Gradient Descent
6. Momentum based GD
7. Nesterov accelerated gradient
8. AdaGrad
9. RMSProp
10. Adam
11. Initialisation
12. Zero
13. He
14. Xavier
15. Activation functions
16. Sigmoid
17. Tanh
18. Relu
19. Leaky relu
20. Softmax
21. Strategy
22. Batch
23. Mini-batch
24. Stochastic
25. Regularisation
26. L2 norm
27. Early stopping
28. Addition of noise
29. Normalisation
30. Drop-out

 All these six categories are essential in building a network and improving its accuracy. Hyperparameter tuning can be done in two ways

1. Based on the knowledge of task
2. Random combination

The first method involves determining the items based on the knowledge of the task to be performed. For example, if classification is considered then

• Activation function- softmax in o/p and sigmoid for rest
• Initialisation- zero or Xavier
• Strategy- stochastic
• Algorithm- vanilla GD

The second method involves the random combination of these items and finding the best combination for which the loss function is minimum and accuracy is high.

Hyperparameter tuning would already be done by researchers who finally report the correct combination of items for maximum accuracy.

HAPPY READING!!!

The previous article gave some introduction to the networks used in deep learning. This article provides more information on the different types of neural networks.

In a feed-forward neural network (FFN) all the neurons in one layer are connected to the next layer. The advantage is that all the information processed from the previous neurons is fed to the next layer hence getting clarity in the process. But the number of weights and biases significantly increases when there is a large number of input. This method is best used for text data.

In a convolutional neural network (CNN), some of the neurons are only connected to the next layer i.e. connection is partial. Batch-wise information is fed into the next layer. The advantage is that the number of parameters significantly reduces when compared to FFN. This method is best used for image data since there will be thousands of inputs.

In recurrent neural networks, the output of one neuron is fed back as an input to the neuron in the previous layer. A feed-forward and a feedback connection are established between the neurons. The advantage is that the neuron in the previous layer can perform efficiently and can update based on the output from the next neuron. This concept is similar to reinforcement learning in the brain. The brain learns an action based on punishment or reward given as feedback to the neuron corresponding to that action.

Once the final output is computed by the network, it is then compared with the original value, and their difference is taken in different forms like the difference of squares, etc. this term is known as loss function.

It will be better to explain the role of the learning algorithms here. The learning algorithm is the one that tries to find the relation between the input and output. In the case of neural networks, the output is indirectly related to input since there are some hidden layers in between them. This learning algorithm works in such a way so as to find the optimum w and b values for the loss function is minimum or ideally zero.

The algorithm in neural networks do this using a method called backpropagation. In this method, the algorithm starts tracing from the output. It then computes the values for the parameters corresponding to the neuron in that layer. It then goes back to the previous layer does the computations for the parameters of the neurons in that layer. This procedure is done till it encounters the inputs. In this way, we can find the optimum values for the parameters.

The computations made by the algorithm are based on the type of the algorithm. Most of the algorithms find the derivative of a parameter in one layer with respect to the loss function using backpropagation. This derivative is then subtracted from the original value.

Where lr is the learning rate; provided by the user. The lesser the learning rate, the better will be the results but more the time is taken. The starting value for w and b is determined using the initialization.

Method	Meaning
Zero	W and b are set to zero
Xavier	w and b indirectly proportional to root n
He	w and b indirectly proportional to root n/2

 Where n; refers to the number of neurons in a layer. These depend on the activation function used.

The derivative of the loss function determines the updating of the parameters.

Value of derivative	Consequence
-ve	Increases
0	No change
+ve	Decreases

The derivative of the loss function with respect to the weight or bias in a particular layer can be determined using the chain rule used in calculus.

HAPPY READING!!

The previous article gave a brief introduction to deep learning. This article deals with the networks used in deep learning. This network is known as a neural network. As the name suggests the network is made up of neurons

The networks used in artificial intelligence are a combination of blocks arranged in layers. These blocks are called an artificial neurons. They mimic the properties of a natural neuron. One of the neurons is the sigmoid neuron.

This is in general the formula for the sigmoid function. Every neural network consists of weights and biases.

Weights- The scalar quantities which get multiplied to the input

Biases- the threshold quantity above which a neuron fires

Notation	Meaning
X	Input
Y	Output
W	Weight
B	Bias

Working of a neuron

This is the simple representation of a neuron. This is similar to the biological neuron. In this neuron, the inputs are given along with some priority known as weights. The higher the value of the weights, the more prioritized is that input. This is the reason for our brain to choose one activity over the other. Activity is done only if the neuron fires. A similar situation is seen here. The particular activity is forwarded to the next layer only if this particular neuron fires. That is the output must be produced from the neuron.

Condition for the neuron to fire

The neuron will produce an output only if the inputs follow the condition.

As mentioned before, the bias is the threshold value and the neuron will fire only when the value crosses this bias. Thus the weighted sum for all the inputs must be greater than the bias in order to produce an output.

Classification of networks

Every neural network consists of three layers majorly: –

1. Input layer
   1. Hidden layer
   1. Output layer

Input layer

The input layer consists of inputs in the form of vectors. Images are converted into 1-D vectors. Input can be of any form like audio, text, video, image, etc. which get converted into vectors.

Hidden layer

This is the layer in which all the computations occur. This is generally not visible to the user hence termed as a hidden layer. This layer may be single or multiple based on the complexity of the task to be performed. Each layer processes a part of the task and it is sent to the next layer. Vectors get multiplied with the weight matrix of correct dimensions and this vector gets passed onto the next layer.

Output layer

The output layer gets information from the last layer of the hidden layer. This is the last stage in the network. This stage depends upon the task given by the user. The output will be a 1-D vector. In the case of classification, the vector will have a value high for a particular class. In the case of regression, the output vector will have numbers representing the answer to those questions posed by the user.

The next article is about the feed-forward neural network.

HAPPY LEARNING!!

Have you ever wondered how the brain works? One way of understanding it is by cutting open the brain and analyzing the structures present inside it. This however can be done by researchers and doctors. Another method is by using electricity to stimulate several regions of the brain. But what if I say that it is possible to analyze and mimic the brain in our computers? Sounds quite interesting right! This particular technology is known as deep learning.

Deep learning is the technique of producing networks that process unstructured data and gives output. With the help of deep learning, it is possible to produce and use brain-like networks for various tasks in our systems. It is like using the brain without taking it out.  Deep learning is advanced than machine learning and imitates the brain better than machine learning and also the networks built using deep learning consists of parts known as neurons which is similar to biological neurons. Artificial intelligence has attracted researchers in every domain for the past two decades especially in the medical field; AI is used to detect several diseases in healthcare.

Sl.no	Name	Description	Examples
1	Data	Type of data provided to input	Binary(0,1) Real
2	Task	The operation required to do on the input	Classification(binary or multi) Regression(prediction)
3	Model	The mathematical relation between input and output. This varies based on the task and complexity	MP neuron(Y=x+b) Perceptron(Y=wx+b) Sigmoid or logistic(Y=1/1+exp(wx+b)) *w and b are parameters corresponding to the model
4	Loss function	Kind of a compiler that finds errors between the output and input (how much the o/p leads or lags the i/p).	Square error= square of the difference between the predicted and actual output.
5	Algorithm	A kind of learning procedure that tries to reduce the error computed before	Gradient descent NAG AdaGrad Adam RMSProp
6	Evaluation	Finding how good the model has performed	Accuracy Mean accuracy

Every model in this deep learning can be easily understood through these six domains. Or in other words, these six domains play an important role in the construction of any model. As we require cement, sand, pebbles, and bricks to construct a house we require these six domains to construct a network.

 Now it will be more understandable to tell about the general procedure for networks.

1. Take in the data (inputs and their corresponding outputs) from the user.
2. Perform the task as mentioned by the user.
3. Apply the specific relation to the input to compute the predicted output as declared by the user in the form of model by assigning values to parameters in the model.
4.  Find the loss the model has made through computing the difference between the predicted and actual output.
5. Use a suitable learning algorithm so as to minimize the loss by finding the optimum value for parameters in the network
6. Run the model and evaluate its performance in order to find its efficiency and enhance it if found less.

By following these steps correctly, one can develop their own machine. In order to learn better on this, pursuing AI either through courses or opting as a major is highly recommended. The reason is that understanding those concepts requires various divisions in mathematics like statistics, probability, calculus, vectors, and matrices apart from programming. 

       

HAPPY READING!!