Hey Everyone ! Welcome to the Semicolon
In this tutorial, we will be implementing a Convolutional Neural Network
using keras with Tensorflow as backend
We use the MNIST hand written digit dataset
to build a digit classifier
If you do not know what a convolutional neural Network
is, or have any questions in the functioning of
a CNN, watch the video about
working of Convoltional Neural Network and that should give you a
basic idea of Convolutional Neural Network
and then we can start.
So, this channel also has lots of videos on Machine Learning
and you might want to subscribe for that
because I have new videos coming up
So let's get started with implementation,
we had run random forests algorithm
on the same dataset
and that gave us around 96% accuracy
So let's see what Convolutional Neural Networks
Offer us here
I import the Libararies, Sequential
from keras.model, also we need to import
Dense, Dropout, Activation and Flatten
from keras.layers
I'll explain you what each of them mean as we use them
For the convolution layer, let's also Import
Convolution2D and for Max Pooling layer Let's
MaxPooling2D
So to validate our model, we'll
be needing train test split so,
I'll be using the one from the scikit learn, because I am very comfortable with it.
otherwise Keras has a utility to do the same function
We need to import pandas and numpy as well
The data is available in csv
and I will import it using pandas. So,
This is our data, this is what it looks like
we'll have to modify a bit to be able to feed it
into our convnets
So, for now all the pixels are arranged
as rows and we need an Image to be represented as matrix to
work on it, so let's reshape the
image, the array using the
reshape function, so I reshape it to
28 X 28 X 1
28 - Height, 28 - Width, 1 - Channel
In the RGB Images, we can have 28 X 28 X 3
It's Black and white Images here so we have 28 X 28 X 1
So I create 2 lists, one is to store
the images and the other one is to store the labels
say if the first element
of df_x contains the pixel
array for image
representing say '4'
then the first element of y would contain 4
So that is why we have 2 lists, one to store the pixel arrays and other
to store their labels
So now, let me go ahead and pull all the pixels
into the list converting them into arrays
and lets store the labels too
now the problem we are looking here is
classification problem, because we have
the digits and we have to classify them into 10 classes
1 to 10
0 to 9 actually, so we need our labels
or results to be categorical
currently they are represented as
0,1,2,3 .....,9 and this is a problem for us
and why is it a problem ?
Because first, labels 1,2,3,...
Signify a relationship
2 * (1) = 2
which is not something we want our model to infer
We are only interested in identifying the
digits in this problem
and we no it can only be recognized by its shape
So shape of 1 representing value 1
is independent of shape of 2 representing value 2
That is why we don't need
any relationship among the labels and that is why we have to make it
categorical. And how does making it categorical
Solve the problem, So
when we make it categorical, we make
different columns for each digit
and then
the digit which is the label
would have value 1 and all other digits would have value 0
So, for example
0 will have vector 1 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0
1 will have the vector 0, 1 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0
So that is how,
these vectors are orthogonal
and they have no relationship among themselves
So, keras has a to_categorical function
which takes in our labels and the number of classes
and transform it to categorical labels
And once we are done, let's convert everything
to the numpy arrays and check the shape
and the shapes are good
then we split the data into test and train and we are done with
preprocessing our data for
for feeding it into CNNs or Convolutional Neural Network
Now let's start implementing the main part
Let's start implementing our
convolutional nerual network model in keras
we need a sequential model
so we start a sequential model
by typing model = sequential() And before we add any layer
let's define our architecture first
we'll then need to test it out and make changes
in it for improvements in accuracy
So we need a convolutional layer, which will
take 28 x 28 images as input
and let the convolutional layer have
have 32 Filters, each of size 3X3
and let's take activation as ReLU
so that we won't have to add normalization layer separately.
And let's add max pooling layer of size 2X2
and then flatten it up into a neural network
let the next layer of the neural network have 100 nodes
and the output layer
of the neural network will have 10 nodes
Since we have converted our input labels to categorical labels (10 dimensional vector)
So let's start by typing
model.add(Convolution2D) , then write the number of filters which is 32
and the size of filters which is 3
and we just have to write it once because we want 3X3 , the length and width of the filter are same so we'll have to write 3 just once
the data format needs to be channel last
since we have rearranged
to be 28X28X1
and activation is ReLU as we discussed
and for the first convolution layer
we need to give it the input shape
so our input shape is 28X28X1
the 1 at the end is the channel
we have to add it because
keras need's it like that, so we add it
otherwise 28X28 would be sufficient
and after this let's add
a maxpooling 2D layer
of pool size 2X2 and then we flatten it
so that we get a fully connected network
and let the next layer have 200 nodes
by typing dense, so dense is a neural network
and what flatten does is it
converts all our matrix to a single vectors
so we'll have all those nodes
in a neural network
and then, let the output layer have
10 nodes, since we need 10 outputs (each output represents probability of digit 0-9)
In the middle, we add drop out
as it helps us in avoiding overfitting
the dropout drops 50% of the weights
in the learning process when we dropout 0.5
then we compile it with loss function
categorical cross entropy
Loss function is something which will calculate the errors
in the prediction
Choosing which loss function
to use is a huge research area in itself
and we will look into it later
the optimizer is something which updates the weights
and that is adadelta for now
we can also look at which one to choose later on
and while learning, we need to see our
accuracy, so in metrics, I specify
accuracy. Typing model.summary()
gives me the summary of the model, where we can confirm
the input and output shapes, so
Input we are taking 28X28 images
and output we are getting 26X26 image
because 3X3 filter and the border
are zeros
so we get 26X26 and we have 32 filters, so we
we get 26X26X32 size matrix
and then when we flatten it
we get a huge number of nodes
then 100 nodes and then 10 nodes
so that is our model and we have checked it
Now let's train the model. we use model.fit()
and pass the training data and for validation,
I pass test data
let's first train the model with just 100 images
and check if its any good
So I run this and the accuracy seems to
be very poor and... what have we missed.
pause it and think about it for a second, because I have figured out what we have missed
The problem at hand is a classification problem
we need the output to be the probabilities of the digits
and to do that we will need a softmax activation
and we haven't done that.
Now just look at the power of the activation function
and how much it can effect the accuracy
I have applied the softmax activation and trained the model again and look at the results
with just 100 digits. Because
there has been an improvement
let's train it on the full data
and this will take around 10 mins on my slow old laptop
If you are gonna do a lot of Machine Learning stuff
specially deep learning stuff, then you better buy a machine
with Nvedia GPU, you can check out some of them
in the description below.
Alternative solutions will be using AWS
or Google Cloud Services
because you won't be able to use your laptop
till this thing is complete
so as this goes on, I want you to have a look at
loss and the accuracy , now if you observe loss
it is continuously falling
and Ideally, we need to run the model till
the loss doesn't change anymore
That is one of the oldest rules in Machine Learning
also, as the loss decreases, the accuracy
keep on increasing as the loss drops
Looks like the accuracy is almost 97 to 98 % which is pretty nice
which is better than the Random Forests
So we can tweak a few things in the model
like the activiation, the number of filters
or the number of layers, the number of convolution layers
order of layers, anything !
that is how we are gonna tweak and improve the accuracy
you can use model.predict() for
predictions and model.evaluate()
for loss and accuracy values
some key points to note is the model has around 500,000 training parameters
which may overfit very easily
because of the high number of parameters
So, be careful with that
By adding
dropouts or using appropriate activations
whenever possible
also
mmm
proper activations and loss functions can
help improve accuracy by a lot
and that is where you need to spend your time on
So that is it, we have successfully implemented
our convolutional neural network
which classifies Handwritten digits upto around
97 % of accuracy, now you can go ahead and
win you million $ competition on Kaggle
Thank you