Hi this is Jeff Heaton, welcome to Applications of Deep Neural Networks
with Washington University. In this video we're going to see how to actually
extract the weights from a neural network created by Keras and put those
weights into an equation so that we can actually calculate the output from the
neural network and see that this there's really no magic to this process, that it
is simply weights multiplied against the inputs that produces a final output. For
the latest on my AI course and projects click SUBSCRIBE and the BELL next to it
to be notified of every new video. This is code from Keras and TensorFlow, but
this applies to really just about anything. I'm going to show you how to
actually extract the weights from the neural network so that we can then put
them onto a diagram and actually actually calculate it and come up with
the same number that TensorFlow would have for this. We're going to use an
exclusive-or neural network. That's the the XOR function, these are the inputs to
it. [0 0] the usual truth table for any sort of an and or an or this is the
expected output. The thing to remember with XOR is if the two inputs are the
same, it's going to be zero. If the two outputs are different 0 & 1 vs. 1 0, it's
going to be 1. Here we set up a neural network, it has two inputs and a hidden
layer of two. We're going for the absolute smallest neural network, just to
show that you don't really need that much to calculate an exclusive-or. Also,
since we are going to calculate this by hand, we're just being lazy. We don't, I
don't want to give you a truly deep deep neural network and then calculate it by
hand. That would not be fine, so we're going to optimize it with mean square
error and the final output is going to be one neuron. It's a regression
neural network, you can do XOR is classification or regression, in this
case I'm doing it as regression. I basically trained it. Here, I am training
it for a hundred thousand epochs. It may take longer because
there's so few weights in this. That your initial random values of your weights
are really going to have a lot of determination
the success of your training. We could do this and way fewer than this many epochs,
this is basically grilling a cheese sandwich with a jet engine, but I am
really not trying to show you how to tune these things right now, I just want
to get the weights set up so that they're good for a exclusive-or network.
We're gonna put those on a diagram and show you that basically you can then
calculate the same output, so you can see the weights behind what Keras gives you.
I trained it. I predict it this is good for the for the neural network because
each of these four correspond to these four up here in the input, this is
scientific notation, so to the -4 that's zero point zero, zero, zero that
number so the these two are effectively 0 and these two in the middle are
effectively 1, so this may look weird if you're not used to seeing numbers like
this, but this is a great output. The, the neural network has trained quite well, so
then what we're able to do is, I write, I wrote this little program here, this
dumps the weights, so you're seeing that the layer 0 counting starting at 0, the
weight from the bias, to the next layer layer 1, again we're counting with 0,
neuron one, neuron neuron zero, neuron 1, these are the biases and these are the
actual weights so there's not a lot of weights and biases in this neural
network, it's it's a fairly small one so I'm gonna show draw a diagram here in
just a minute, and we're going to copy all of these values up onto the diagram,
then we're going to calculate it by hand. I give you the code down here to
calculate it by hand as well, if you want to do it you set the input to 1, and 0, so
1 and 0, and an XOR those are different, so it should be 1, so it outputs a 0.96.
Now since we're calculating this by hand i truncated quite a few of these numbers
so 1.29 becomes 1.3 and we'll do that when we diagram it so we lose a little
bit of accuracy, but these are the two input neurons, we are going to feed this
values 0, and one so that's what we're calculating
for, you could calculate it really for just about any other value that you
wanted to, but for the other four that you would have in the exclusive-or. We're
also going to have the bias neuron, so like we saw in earlier class videos,
you're essentially calculating a weighted sum over and over and over,
again for a neural network, so we're going to calculate the value for hidden
or on 0 ,H0, H0 has inputs from those previous 3.
These all have weights as we saw from the output from the program before. So,
I'm just going to copy those in
so this is going to become essentially a weighted sum, that we're going to use to
calculate what hidden zero is actually, what its value actually is, so to do that
we're going to multiply the input zero times its weight which is 1.3,
plus 1 times 1.3, that one is from the first which is
actually the second input neuron, and then we have to add the bias, and we
basically perform that calculation, which ends up being zero, so the value for
hidden in hidden neuron 0 is zero. Next, we're going to calculate the value
for hidden or on one which is the second one has a similar sort of thing going on,
it has three weights coming into it. It's going to copy those values from the
output that we had from the Keras program so that is simply 1.2,
we are calculating basically, the weighted sum again the first input I0
row drops out second one contributes that 1.1, 1.2 to it and
then the intercept, or the bias, for, for this for the second hidden neuron is
zero, so this whole thing together is basically equal to 1.2, then
this value we have to pass it through the through the activation function both
of these needed to go through the activation function, that is the
rectified linear unit, so that is basically going to be the max of 0 and
1.2, which is also 1.2, so the output from this hidden
neuron here is 1.2, same sort of deal up here to apply its activation
function, it's going to be max because it's also the value in max zero. Zero is
zero, the activation functions that you can
use, I may give you a different one for the for the midterm examination on this
one, the only other ones that you will ever see at least in this class are
sigmoid, hyperbolic tangent there are others but these are for regression if you're
doing a classification, you might also see the softmax, but focus primarily on
these now. The output the final layer, so that we actually get what this this
neural network is providing for us.
This is going to have similar to before. We do have another bias connection so,
you're going to have essentially three and in-bound connections again and we're
essentially going to just copy the weights that we had from the output in
Keras, there is zero and this results and the final equation that ties this
whole thing together and that is going to basically be the output from, from,
hidden hidden 0 times 1.6 plus 1.2 times 0.8 and
see the 1.2 times the 0.8 and then plus, the plus the
the bias or the intercept which is zero, and then this whole thing equals 0.96,
that's your final output. That is approximately equal to 1.0 which
is what you would have expected from a exclusive-or of zero and one there you
have it that is the calculation for this neural network, done using the weights
that Keras is actually trained for us, so we can see that there is really no magic
there, this simple of a neural network you could have actually just hand coded
these weights that would have not been difficult. I'll probably do a video on
that as well so that you can see how to actually hand code one of these you do
need about this size of neural network. You definitely need the hidden layer.
Thank you for watching this video, in the next video we're going to see more
advanced training techniques that we can use with a neural network to to get
better results, and to measure our error in a variety of different ways. This
content changes often, so subscribe to the channel to stay up to date on this
course and other topics and artificial intelligence.