Hi, I’m Adriene Hill, and Welcome back to Crash Course Statistics. In previous episodes
we’ve talked about things like cars learning how to drive themselves...and apps that can
recognize handwriting and turn it into printed text.
A lot of these projects are done using a type of Machine Learning called a Neural Network.
The term Neural Network covers a bunch of different--but related--methods that can take
in data and spit out useful outputs.
Neural networks can output everything from the probability of someone getting a particularly
nasty strain of MRSA on their next hospital stay, to new chapters of Harry Potter...seriously.
They may even be behind some of the annoying Twitter bots that just seem to spout tweets
that rile people up. Today, we’re going to take a look at the
big picture of what neural networks are, and how they do all these things.
INTRO
In Crash Course Computer Science, we talked a little bit about what a neural network is.
In the simplest sense, a neural network looks at data and tries to figure out the function--or
set of calculations--that turns the input... variables...into the output.
That output could be a number, a probability, or even something a bit more complicated.
Neural networks are analogous to robots that can learn to make things---like a toy car--not
by following step by step instructions from humans, but by looking at a bunch of toy cars
and figuring out for itself how to turn inputs (like metal and plastic) into outputs (the
toy cars)!
If we want to work with data instead of toy cars we can use a neural network to predict
future salary based on a number of variables such as degree, field, age, years of experience,
gender, number of promotions, and university.
We feed these variables to the neural network. These circles are called Nodes, and they just
hold a value like degree or field.
Eventually we want the Neural Network to output its prediction for future salary. So we know
there will be one output node at the end of our network that tells us what it predicts
the salary will be .
At this point, the Neural Network looks kinda like a regression, we have a bunch of inputs...our
variables...which are combined in some way to create an output...our predicted value.
But unlike most regressions, neural networks feed the weighted sum of age, degree, field,
etc through something called an “activation function” which takes the value and transforms
it before returning an output.
These activation functions improve the way many neural networks learn, and give them
more flexibility to model complex relationships between input and output.
One common activation function is called Rectified Linear Unit (ReLU) --which turns all negative
values to 0, and leaves positive ones as they are.
This makes these nodes act a little bit like neurons in your brain--hence the name neural
network-- which require a certain “threshold” of activation before they’ll fire. So a
node with 0 doesn’t fire, or contribute to the output at all. But one with a positive
value will.
This Neural Network currently has two layers--input and output. But we can add layers between
them. So now the inputs are indirectly connected
to the output, through the middle layer of nodes.
It’s pretty clear what the input nodes are, since they’re values we understand. And
the output node is a salary, so we get that too. But it can be harder to grasp exactly
what the middle layers represent.
You can think of all the calculations that happen between the input nodes and output
nodes as something called “feature generation”. “Feature” is just a fancy word for a variable
that can be made up of a combination of other variables.
For example we could use your grades, attendance, and test scores to create a “Feature”
called Academic Performance. Essentially the neural network is taking the variables we
give it, and performing combinations and calculations to create new values, or “features”.
Then, it combines those “features” to create an output.
When we have large amounts of complex data, the neural network saves us a LOT of time
by combining variables and figuring out which ones are important. Neural Networks allow
us to make use of data that might seem too big and overwhelming for us to try to use
on its own. They can find patterns that humans might never be able to see.
If a neural network has more than one layer, we say that we’re using “Deep Learning”,
since there are many layers of nodes. Deep Learning has gained popularity in recent years.
Neural networks and deep learning have been used extensively to do things like recognize
handwritten numbers and simulate x-ray images so airport security can be trained to recognize
items like drugs and guns.
There’s a lot more math that goes into neural networks . But in short, they learn by figuring
out what they got wrong, and then working backwards to determine what values and connections
made the output incorrect.
For example, if it predicted my salary and is $10,000 off, it will take that difference
and figure out which parts of the neural network were influential in creating that $10,000
error. It then tweaks them so that next time, it’s not as wrong.
You can see that in this neural network--sometimes called a Feed Forward Neural Network--all
the nodes only feed into the next layer from input to output. Hence, they only Feed information
Forward.
But it is possible to feed the output of a Neural Net back into the model as an input
the next time you run it. In other words, nodes in one layer can be connected to each
other, even themselves! These types of Neural Networks are called Recurrent Neural Networks.
We can use RNNs to learn patterns. For example, words! RNNs have been used to spell check
text. The Network can learn to take in a misspelled word like this... and correct it.
Often we use this kind of network when we have sequential data-- like stock prices over
time, or the words in a sentence. If you’re trying to predict the words in a sentence,
it matters a lot what the previous word was.
If the previous word was “A”, that influences what the current word is. Usually the word
“A” precedes a noun, or an adjective -- one that starts with an consonant. A Fox. A Quick,
Brown Fox. But it’s unlikely to precede a verb. “A walked” wouldn’t make sense.
But the further you get through the sentence, the less influence the word “A” has.
Unlike Feed Forward Neural Networks, Recurrent Neural Networks “ remember “ the previous
outputs. For example, if we used a Recurrent Neural Networks to generate a melody, we would
give the network some information about our song framework, and we’d ask it for a note.
Then we feed that note back into the model along with the information about our song
framework and the network would generate the next note.
In order to make a melody that sounds good, the Recurrent Neural Network needs to “remember”
what the previous notes were. Using the outputs as inputs allows us to do that.
A popular type of Recurrent Neural Network called a Long Short-Term Memory Network has
been used to generate all kinds of music. It’s even been used to write a few new Harry
Potter chapters.
ahem Here is one of those chapters from a Recurrent Neural Network trained by Max Deutsch…
“The Malfoys!” said Hermione.
Harry was watching him. He looked like Madame Maxime. When she strode up the wrong staircase
to visit himself.
“I’m afraid I’ve definitely been suspended from power, no chance — indeed?” said
Snape.
He put his head back behind them and read groups as they crossed a corner and fluttered
down onto their ink lamp, and picked up his spoon.
The doorbell rang. It was a lot cleaner down in London.
So, J.K. Rowling isn’t out of a job yet. This excerpt doesn’t make sense within the
context of the Harry Potter universe, or really make sense at all. But it at least has the
structure of a book chapter.
We can also use Neural Networks to look at another form of art: images. A lot of applications
of image recognition use a type of Neural Network called a Convolutional Neural Network.
Images are made up of a grid of pixels.
A very tiny grayscale image like this could be represented by a grid like this ...where
each number represents how much black is in that pixel. 0 is complete black, 1 is complete
white, and anything in between is a shade of gray.
Color images are a little more complicated, since each pixel has a red, green, and blue
value, but the idea is similar.
In this case, a pixel is affected by all the pixels surrounding it. It’s not simple sequential
data. So, convolutional neural networks look at “windows” of pixels instead of one
pixel at a time.
They apply a filter to these windows to create “features”. This step is called convolution.
The filters that the network uses are just calculations that transform the pixels that
are inside the window. The network uses the data to determine which windows and filters
will be used.
Some filters might help detect edges in the image
Others might recognize features like curves, horizontal lines, or even more complex objects
like eyes, or faces. These features make it so we can take an image...which has a huge
number of pixels...and make a smaller number of features.
This process is called pooling. In the end, the network will use the features generated
by convolution and pooling to give us some kind of output, like a decision about whether
or not an image contains a stop sign, or a human face.
Snapchat, for example, has used variations of convolutional neural networks in their
app. And these networks are used extensively in all kinds of image recognition.
If you hate those CAPTCHAs that ask you to click on each image that has a stop sign,
you could use a convolutional neural network to fill them out for you.
And the next time you’re in another country, you can use Google’s Translate app which
uses these networks to help translate the text from signs or menus into your language.
One thing that limits our use of neural networks of all kinds is a lack of data. The more complex
these networks are, the more data they need to perform well.
But some neural networks can be trained to generate data. These are called Generative
Adversarial Networks (GANs). They use sets of existing data to try to learn how to create
new data. These networks are kinda like two neural networks...disguised as one...by wearing
a trenchcoat.
We’ll illustrate how they work, with an analogy. Let’s say you’re a counterfeiter
who’s trying to make fake $100 bills. You examine a few $100 bills create a fake
and then try to use it at your local convenience store. If the bill is rejected, you politely
ask the cashier what made them realize the bill was fake. And they’re happy to help.
They tell you, you take this information back to your counterfeiting lab, and make a new,
better fake $100 bill.
You repeat this process over and over--hopefully the cashiers don’t start to recognize you...and
eventually, you should have a passable fake bill. (Assuming you aren’t already in jail.)
However, since the cashiers are seeing so many fake bills, they get better at recognizing
them as time goes on.
In our analogy, you are the generator. Your job is to make fake input...in this case $100
bills that are good enough to “trick” the convenience store. The cashier is the
discriminator since her job is to learn to discriminate between real and fake $100 bills.
Essentially you have two neural networks battling it out to create better and better outputs.
The generator is trying to get better and better at making data that can trick the discriminator.
And the discriminator is trying to learn how to best discriminate between fake and real
data.
These networks have been used to create new anime characters , make new Van Gogh-like
art, and create new skate decks
Neural Networks of all kinds help us deal with the big, sometimes messy data that we
have in real life. They help detect patterns in data that humans can’t see.
And often, they’re way better than we are at figuring out how to turn input variables
into accurate outputs. While you and I don’t have time to look through Terabytes of data,
a Neural Network does. With neural networks we can make use of data that may have been
too overwhelming without them.
We’ve talked about Neural Networks being applied to a lot of different things, like
image and face recognition. That type of technology could one day be utilized in drones on search-and-rescue
missions.
Natural Language Processing...often using neural networks...is the reason we have Alexa.
Also, hey Google, hey Siri, hey Cortana, hey Bixbi. I think I got everybody.
As data gets bigger and bigger, neural networks will likely become a more common tool for
making the most of our data, and understanding how they work allows us to think of really
creative ways to use them. Hopefully, less annoying ways. Thanks for watching, I’ll see you next time.