[MUSIC PLAYING]
JASON MAYES: In the prior sections,
you took a deeper dive into regression,
which is essentially the art of predicting an output
number from some inputs.
You started with a simple linear regression prediction
using a single neuron without an activation function
and then progressed to predicting non-linear data
using multiple layers of neurons with activation functions.
In this section, you will learn about classification
which is all about predicting what class some input
features represent instead.
Let's start by visually reminding yourself
of what classification is doing.
Essentially, we have some data points
with some features like the ones shown.
If you plotted them on a scatterplot,
it might look something like this.
However, in this case, each point
represents a certain class of data.
Maybe this scatterplot is showing results
for two different types of flowers
plotted against two common input features--
maybe the number of petals and stem length, for example.
Now, in the previous section, you
learned that by combining layers of neurons,
you can essentially combine multiple lines
of best fit to create a system that can predict
non-linear data.
In the case of classification, you can do something similar.
But instead of using that line to predict a single number,
you use it to separate one class from another, as shown.
Here, you can see how three lines were used to separate
the orange from the green dots, which cannot be separated with
a single straight line.
And what about this slightly more extreme example
with a lot more data?
It's the same principle, really.
And with enough neurons and layers,
you can potentially train a system
to learn some very complex and intricate classifications.
In the example shown here, for a new given
recording, if it's contained within these lines,
it's a class II.
And if it's contained outside, then it's going to be class I.
Now, what if you had an example point for every x and y value
in the graph within some given range?
What would that look like?
Well, an image actually fits that description pretty well.
Here, you've got a magnified gray-scale image
with the number one drawn on it in white.
Each pixel is represented by a colored square in x and y
on the graph, as you can see.
And just like before, you can define
what parts of the image the number 1
is typically contained in.
So for all new images, you can check all the non-black pixels.
And if you find they're mostly contained
within the same bounds of these three lines,
then you have seen a number 1.
And contrast this to, say, in number 5,
and you can see how many of the white pixels
are outside the bounds of the bound lines for the number 1,
with only 30% inside.
From this, you can see from the high level
at least how you might be able to train a system to tell them
apart, with neurons looking for different features
in different parts of the image.
So maybe you want to recognize handwritten numerical digits
like this.
You would have 10 possible numbers.
You would need to recognize 0 to 9.
Let's see how you could create a simple model to do that.
First, though, how might you break down an image
to be fed into your model?
Well, it's actually pretty easy to break down
into the rows of individual pixels
that make up the image as shown.
After all, a gray-scale image is just
a matrix of numbers ranging from 0 to 255,
representing different shades of gray, from pure white being 55.
And pure black being 0.
Here, you have an example image that's just 28
by 28 pixels in size.
So it would have 28 rows.
You now take all of the rows of data
and turn them into a really huge one-dimensional array
of 784 values, which you can then use as the inputs
to your machine learning model.
OK, so let's put it all together.
A 28 by 28 pixel image would represent 784 input values
to your model as shown after all your rows
are broken down into one big long line of numbers.
In this example, you have a gray-scale image.
So these input values represent 255 different shades of gray,
depending on the number that they are.
Next, you could add a fully connected dense layer
of, say, seven neurons.
And every neuron in this input layer
samples all of the 784 input pixels.
Here, you can see all of the first neuron's connections,
shown by the orange lines, for example.
That means there are 5,488 connections made
between those seven neurons in every pixel input,
even though this image is tiny.
You can see how larger images would get very computationally
intensive very fast, which is why
image inputs in machine learning are typically pretty small.
Next, you can add a hidden layer of five neurons that
sample the outputs of the neurons
in the input layer shown here, which creates a further 35
connections, as these are also fully connected.
Finally, you can add 10 more neurons in the output layer
that sample the outputs of the hidden layer,
creating 50 further connections.
Note that your neural network now has
a total of 5,573 connections.
That's a lot of weights to train.
Also, having 10 outputs is different to what
you had done before.
In this case, each neuron in this output layer
represents a vote for each one of the possible 10 digits
from 0 to 9, which is why there are
10 neurons in this last layer.
In this situation, once all the weight calculations
have flowed through the network, the output neuron
with the highest value is the one
it predicts is the correct class for the input image.
You can also see how your networks of neurons
are getting larger and with time may even
have more layers, which is where the term "deep neural network"
actually comes from, as they have a high layer depth.
Assuming numbers are drawn roughly centered in the input
image, you can imagine at this stage
that by adjusting over weights and biases in your model
thousands of times, it will slowly
be able to figure out that for a given digit,
certain pixel inputs are more important than others
and give those pixels higher weights.
Remember, all of the weights are initially random.
So it'd be pretty bad at the start.
But if it gets something wrong, it
can calculate how wrong it was and update the weights
to move in the right direction.
So we tend to produce at a higher
value at the correct output that represents that digit.
But how does it know which output represents which digit?
If you just take your current code to have 10 outputs,
it will not work as expected.
Instead, you must turn the network
into one that classifies instead of performing regression.
Let's see how to do that.
First, you must learn about a new technique
known as 1-hot encodings.
When dealing with categorical data,
such information can be represented
numerically in a fair manner.
The name 1-hot sounds pretty random.
But the principle is pretty simple.
Currently, you're trying to predict 10 possible digits
for numbers from 0 to 9.
However, if you're using the literal numbers as the target
output, here, the model may think
that a 9 is worth more than a 0, which is not the case.
Or what if I predict the value of 3.5?
What does that even mean when you're
trying to deal with images that represent whole numbers?
Also, consider a different use case
where you're not trying to predict images of numbers.
But instead, maybe the images are of cats and dogs.
How do you represent those numerically in a fair way?
Remember, after all, you just have a bunch
of weights and biases you're trying
to tweak to get some numerical answer, not text answer.
Well, with a 1-hot encoding, you simply
have a one-dimensional array of numbers
where all the numbers are zero, apart
from the position that represents the class
of interest, which will be a 1.
You can see here that the number 4 is represented
by the 1-hot encoding shown where
the fifth element in the array is sent to 1,
as the first element would represent the digit, 0.
It's zero-indexed after all.
In a similar fashion, you can convert all of your output
answers into 1-hot encoding.
So instead of predicting one output,
you'll be predicting 10 outputs, where just one of those outputs
is set to 1.
The machine learning model can then
learn to produce the highest value in that position
for that example, image data.
And then that becomes a prediction
of what it thought it saw in the input image.
So putting it all together then, you
can see now how the outputs used to train the system
are represented by this 1-hot encoding.
Ideally, in the situation shown, with a number 9 as an input,
you want to produce an output where
the last element in the output tensor is close to 1.
And all the others are close to 0.
You then know that the network thinks it saw a 9.
So the next piece of the puzzle is a new activation function
known as softmax.
Currently, all the models you've had,
had no activation function on the output layer,
as you were just performing regressions.
In order to perform classification instead
of regression, you will need to use a special activation
function known as softmax.
Again, it's another fancy name.
But all it means is that it will just produce output numbers,
no matter how many outputs there might be, that sum to 1.
The only other thing to note is that softmax will only
work well if the outputs that you're trying to predict
are mutually exclusive, that is, that they're never
overlapping with each other.
For example, any given number can't be both a 1 and a 2.
It's either but not both.
If you know that the hand-drawn number is a 2,
you know for sure that the number can't be anything else.
Now, that sounds pretty good for your classification problem,
as you want to know how sure it was when it saw something,
with very tiny values for the wrong classes and a value close
to 1 for the class that is most likely.
It seems like a perfect match for your 1-hot encoding.
Now, the final piece of the puzzle
is a new loss function that you can specify.
It turns out that different loss functions work best
for different types of jobs.
In the case of regression, you use the mean squared error
that you previously learned about.
However, the multiclass classification
in the case of its digit recognition task,
you will need to change your loss
function to use something called categorical crossentropy.
This is simply a loss function that's
well-suited for classification tasks
with three or more outcomes, like the one you have here.
Even better, TensorFlow.js has it implemented for all of you
to use.
So you can just change the name of a loss function as shown.
And you'll be good to go.
Also, take note, there's also one called binary crossentropy
if you just have two classes to predict in case
you ever need to do that.
For example, you might need to check if an image contains
a hot dog or not.
In this case, there would be only two possible outcomes
if you're only interested in hot dogs.
OK, so you've got the theory covered.
Now, it's time to find some training data.
Remember, you wanted to recognize
all the numerical digits from 0 to 9.
There's actually a famous training data set
of this called MNIST that contains
hundreds of handwritten digits like the ones shown
on the slide.
If you can get that data, you can potentially
train a deep neural network to learn from that data.
Well, in the name of time, I've already got a copy of that data
for you, normalized it, and transformed it
into a JavaScript-friendly array form.
So in the next section, you'll take that data
and write some code to create your first image classifier.
See you there.
[MUSIC PLAYING]