Hi this is Jeff Heaton welcome to Applications of Deep Neural Networks,
with Washington University. In this video we're going to have a general
introduction to deep neural networks and conceptually how they work. This will
allow us to build upon the Python that we learnt in previous parts of this
course and to actually construct neural networks. For the latest on my AI course
and projects click SUBSCRIBE and the BELL next to it to be notified of every new
video. Neural networks have been around for a while. Deep learning is just the
ability to train neural networks that are very deep. You might have seen other
machine learning models and other classes like support vector machines and
gradient boosting and XGBoost and like GBM and all these various ways of
training a model based on data. Neural networks can be drop-in replacements for
other models like this. The neural network simply takes in the data that
you would have normally sent to a support vector machine, or other model.
And outputs either a classification result, where it attempts to classify or
determine the type of what the input was sent to it or it can be regression and
it outputs a number based on the data that you're trying to have the neural
network predict from. However neural networks, their real power lies in their
ability to take in data and produce data in ways that don't fit into your typical
classification and regression type models. For example, you can input an
image into a neural network, you can even have a neural network accept an image
and produce another image. So it gets very expressive, what you can actually do
with the neural network because the input can be just about anything and the
output can be just about anything, and the input/output by no means have to
be of the same type. In other models you would simply send in a one dimensional
vector, so a list of predictors you can also,
though with a neural network pass in a 2D matrix. Now this is where you start to
pass in, say an image with a grid of pixels. The power of the neural network
is that it realizes that pixels that are near each other are more important to
each other. Whereas the other model types changing the order of the input vector
really has no effect on anything it can also pass in a 3D matrix or a 3D tensor
this is would essentially be a color image where the third dimension is
specifying the color of the individual pixels that you're passing in. And
dimensions it can be used in several different ways
with neural networks when you talk about the dimensionality of the neural network.
Usually you're talking about what the input vector or input matrix looks like,
how many input neurons do you have, and how are they arranged. Are they a grid?
Are they a box dimensions? Can also refer to the number of weights that are in a
neural network now traditional models you would talk about regression and
classification, and neural networks do this to the output neurons of the neural
network become either the single regression output or the classification.
So a regression neural network like you see here, now these are two example
neural networks and I put together. I work in the life insurance industry, so
you'll see a number of examples from me sort of in an InsureTech sort of way,
where you've got inputs that are related to medical records and other things that
you'd be interested in for life insurance. Here you're asking the neural
network to predict the maximum face amount, so how much should we insure
somebody for. Ahat's the maximum amount we would go on the risk for with an
individual. Classification neural networks, those produce classes so we
would maybe have a preferred, standard, sub-standard or a decline. That just puts
the the potential insurance applicant into
the correct bucket. So regression, classification, these are your
traditional types of model, and we'll see later on in this class that neural
networks can have much more complicated outputs than just classification or
regression, and neural network can even be both classification and regression at
the same time. The output neurons, so the ones on the far right here, there's just
one if it's a regression neural network, it's always going to be just one output
neuron. Here we have additional output neurons, one output neuron for each class.
That's how you can tell a classification neural network. If it's a binary
classification neural network, meaning it's only classifying between two things,
usually it'll just have one output neuron and that specifies the
probability of it being one of those classes. However, if you're dealing with a
multi-class classification, with with three or more classes, usually you're
going to, you're gonna have one output neuron per class. You would never have a
single class classification neural network, because there's just one class,
it would always be that class. So, you have to have at least two classes so
that there's at least something to differentiate between for the neural
network to classify. This is the structure of a neural network. They have
multiple layers. Now we'll see that there are additional layer types and other
things that that will make this more complicated. But for tabular neural
networks, this is where the input to it looks sort of like rows and columns
from Excel, this is what it'll look like. You're going to have your input layer,
these input neurons are the values that come into the neural network. Then you're
going to have several hidden layers. Finally going to the output layer. These
are bias neurons. You don't send input into those directly. They are simply
there to give the neural network additional
predictive power. We'll see exactly what bias neurons are for in a moment, they
handle the situation where the inputs are both zero, but you don't
necessarily want the output to also be zero. The arrows are the weights between
the various entities, and in the neural network and hidden layers you can have
lots of hidden layers. In deep learning, hundreds of them, it's usually four types
of neurons and a neural network and put neurons take in the input to the rest of
the neural network. Hidden neurons, neither, they're hidden because they're
between the input and output neurons input neurons, receive the input for the
neural network output neurons receive the output that are sent out of the
neural network. Hidden or between that context neurons, we'll see more about
those when we get into time series and recurrent neural networks they maintain
state between calls to the neural network and then bias neurons they are
essentially like the y-intercept in traditional mathematics, linear
equations you also have several layer types that these neurons go into there's
the input layer that receives the input the output layer that sends the output
from the neural network and then the hidden layers between that. In a later
part we're going to manually calculate the output from a neural network, but for
now we'll see that the the calculation that a neural network actually goes
through is not that complex. It's essentially a weighted sum passed
into a activation function. So the input to calculating one hidden neuron, or an
output neuron in the neural network essentially takes in the the feature
vector, or the the vector coming in to it, so if we were calculating for this
neuron here the input vector would be 1, 2, 3
these values and essentially you would multiply input 1 times weight 1, input 2
times weight 2, and put 3 times weight 3. Sum those all together, that
value then gets passed into the activation function and that becomes the
output of that neuron and that's what this equation is basically showing you
here. It's essentially the summation of all the thetas. Thetas, or weights, times
all the X's the X's are inputs and then phi is the the activation function and
this is basically done over and over and over again to calculate every hidden and
output neuron in the neural network. This gives you an example of this the input
is 1 and 2. Now this third neuron, here that's actually your bias neuron and the
way that it gets represented is the bias neuron is we concatenate a1 onto the end
of this. So, 1 is going into one first neuron, 2 goes into the second, and this
one goes into the third neuron. That becomes basically the bias so whatever
this weight is just gets added to it kind of like an intercept since this is
1, we're multiplying 1 times the 3rd weight. That's basically how bias neurons
work. We might have this as our weights so those might be the three
weight values the third one would be called the bias value we multiply each
of these inputs by each of these weight values in summation, and this zero point
eight becomes the summation that is then passed to the activation function.
Activation functions are just functions that introduce non-linearity into the
neural network, so that everything is not linear. Rectified linear unit, we will see
is the most popular of the activation functions in modern deep deep learning.
And other family members related to rectified linear units like the leaky
RELU, you Softmax is all used in classification neural networks for the
final output of the of the neural network. This ensures that all of the
output neurons sum to one. If you'd like those output neurons to be the
probability of each of the classes that the neural networks trying to classify
then softmax just ensures that all those probabilities are truly probabilities
and they add up to one, as probabilities usually do. The rectified linear unit is
a pretty simple activation function but extremely effective you're essentially
taking the max of 0 and X so X is the value that was passed in softmax looks
like this again it's it's essentially summing them together dividing that
dividing each one. By the summation that normalization is what ensures that they
add up all to one point. Oh by the way, if you'd like to experiment with the
softmax and see how these individual values are created I have a JavaScript
example that I have a link to. Right here and why is the rectified linear unit so
popular why is this such a popular activation function used to be before
the deep learning days the most common activation functions were the hyperbolic
tangent and sigmoid sigmoid is shown here so as you would input values into
it it would have the squashing effect as you went to negative infinity or
positive infinity and that was desired behavior. That worked that worked quite
well, you're typically optimizing these neural networks through through gradient
descent so you're taking the derivative of the error function so this is the
error function as we change a weight so as we change one weight.
The error goes, the error goes down you'd like to get
the error at the minimum location, except you can't see this entire graph at once.
You'd have to literally calculate the neural network for every number between
a reasonable range. You only really see the dot that you're actually on but
that's what the derivative is for in calculus you take the derivative and it
gives you the slope, or the instantaneous rate of change of wherever the weight is
at and you can tell by the slope of this value this has a negative slope so that
means we need to increase the weight to move towards the minimum. You're always
changing the weight by the inverse of the sign of the of the slope or the
gradient. This is typically called gradient and machine learning so if we
look at the derivative of the sigmoid function, you can see it by the dot that
the dashed line notice how it quickly converges to zero this since since that
derivative is called a gradient the fact that it would go to zero or vanish
means we would have a vanishing gradient. This is the vanishing gradient problem
in the deep learning solved one of the many problems that it that it mostly
solved because this saturates to zero it's not as desirable as the as the RELU,
you that the rectified linear unit doesn't saturate to zero like the
sigmoid function does in both directions. We'll look at why bias neurons are needed
essentially they are the intercept, like when you previously work with y=mx+b
linear linear equations. If you look at this one this is an example of
when we change the weight if you notice we change the weight it's effectively
changing the slope and you really the line always has to pass through zero.
When the input is so all of these pass through zero there's no way to really
shift that. When you change the bias neuron now
you're shifting, you're not affecting the slope, so using those two together you
can really, you can affect the slope and you can shift it you can move it and
then all of those neurons together can contribute and sort of the additive
effect of all of the neurons. Let's this line then break and and basically
approximate any function. Thank you for watching this video. Now that we've seen
a general introduction to deep neural networks we're ready to start to look at
TensorFlow and Keras, and see how these are actually implemented in Python
so that you can make use of them. This content changes often, so subscribe to
the channel to stay up to date on this course and other topics and artificial
intelligence.