Hello world, it's Siraj.
And today we're going to learn how
to generate videos using the hottest type of model
in machine learning right now, the generative adversarial
network.
We'll train our model to generate the alien language
from the movie "Arrival" that we can later animate.
In 2014 everyone was drenching themselves
in ice water for the ALS challenge,
but meanwhile, in Canada, a researcher
named Ian Goodfellow published a paper
introducing the world to GANs.
And the AI community loved it.
Yann LeCun, the director of AI at Facebook,
called it the most interesting idea in the last two decades
in machine learning.
I recently interviewed Ian and asked him
a bit about its history.
How did you come up with this idea?
(ian goodfellow) For GANs, the short story
is that I was arguing with my friends in a bar.
[MUSIC - 50 CENT, "In Da Club"] [SINGING] Go, go.
(ian goodfellow) The thing that clicked there in the bar
was this idea of having the discriminator continually learn
at the same time that the generator is learning.
If both models learn, they're driven
to this equilibrium where it becomes impossible to fool
the discriminator.
(siraj rival) GANs are a type of generative model.
Given some input data x and some labels y,
it learns the joint probability distribution of this data.
With the joint probability distribution function
given a y, you can calculate or generate its respective x.
And since this is unsupervised learning,
we have no labels handed to us.
We use the training data as y and the generated data as x.
So given say a set of 3D models of major cities
from across the world from Google Earth,
there are thousands of different features or dimensions
to this data.
Can we generate an entirely new city from it?
Atlantis is waiting.
And what constitutes a good city?
Sometimes a single input corresponds
to many different correct answers, each of which
is acceptable.
Generative models help us work with multimodal outputs
like this.
There's been so much activity centered around GANs lately.
They've been used to convert low res images into crisp high res
images, convert hand-drawn sketches
into photo realistic images and to generate everything
from fashion styles to new product categories.
Apple, you should probably take a look into that last one.
BiGANs, cycleGANs, WGANs, there's a lot.
So the basic idea here is that we have two neural nets.
One is called the generator, G and the other
is called the discriminator, D. We got some data set.
Let's say it's a collection of Pokemon images.
And we want to generate new images from this data set.
That means entirely new Pokemon that have similar attributes
to those in our training data.
The generator's job is to try to create
fake Pokemon that look really similar to our training
Pokemon.
Ugh.
The discriminator's job is to classify the generated Pokemon
as either real or fake.
Think of G as a magician and D as his audience.
The magician is constantly trying
to make the audience believe that his illusions are
real magic.
The audience boos when it can tell
that his tricks are fake and applauds when
it can't tell the difference.
They both improve.
And in the ideal case, the magician
gets so good that no matter what he's able
to fool his audience every time.
The type of GAN we'll use in this video
is called a deep convolutional GAN or DCGAN.
This is because both D and G are deep convolutional neural nets.
So the discriminator has several layers
of what's called convolution, and the generator's
got several layers of D convolution.
Convolution is literally impossible to understand
in a parallel universe, but in this one, it's pretty easy.
Normally neural net layers are fully connected.
So all inputs are connected to all outputs.
With many types of data, this makes sense.
We want all parts of the input data
to be able to contribute to all parts of the output prediction.
But images are considered spatially locally correlated.
That means that there's a banana in an image,
it doesn't matter where it is in the image, it's still a banana.
So we exploit that.
Instead of connecting all of the input data, or pixels,
to all output values, we use a much smaller filter
that we slide across the picture like a flashlight,
so that means much fewer parameters
in a convolutional layer as opposed
to a fully connected one.
Let's look at our code.
We're only going to use karas to build our model.
Pillow helps us do image processing,
and NumPy will help us perform some valuable reshaping
operations on our images.
Let's get started by first defining our discriminator.
We'll give it its own function.
It's going to be a linear stack of layers.
So we'll define it as sequential.
And we'll start off with two convolutional blocks.
That means a convolution to extract the feature
map, followed by the tanh activation function
to squash real numbers into a range between negative 1
and 1, which lets our model learn more complex functions
than just linear regression.
Then a pooling layer, pooling reduces the dimensionality
of each feature map but retains the most relevant information.
We'll flatten the feature map into one dimension then apply
two fully connected layers to it.
The last dense layer outputs an n dimensional vector,
where n is the number of classes we have.
So it would be two in our case.
And by applying a sigmoid to it, it
will convert the data into probability values
for each one.
[SINGING] G, A, Ns, got to train them all.
It's D and G. Let's use cross entropy.
G, A, Ns, gradients to descend, is it real or just pretend?
[SPOKEN] Next we'll define our generator, which
performs similar operations but in the reverse order.
Since it's fed random numbers as its input,
it converts them into an image by first going through
to fully connected layers with their own associated activation
functions.
Batch normalization will apply a transformation that
maintains the mean activation close to zero
and the activation standard deviation close to one.
This allows for faster learning and higher overall accuracy.
Than we apply two convolutional blocks that will eventually
output an image.
Upsampling will convert our image into a higher resolution.
Now that we've defined models for both D and G,
we can combine them to make a GAN pretty easily with karas.
We can just reuse the same network objects we've already
instantiated, and they'll conveniently
maintain the same shared weights with the previously compiled
models.
We're going to want to freeze weights in the discriminator
part of the GAN when we back propagate the joint models.
So we'll first set the karas trainable flag
to false for each element in this part of the network.
We'll also define a load data function,
which will form our images into vectors that we
can feed to our model by making use of the image and NumPy
libraries.
OK, so how do we train this thing?
The goal of training our discriminator
is to maximize D of x for every image
from the true data distribution and minimize
D of x for every image not from a true data distribution.
The goal of training the generator, G of z
is to create samples that fool D. We train both D
and G by taking the gradients of this expression with respect
to their parameters.
Because each player's cost depends on the other
player's parameters but each player cannot control the other
player's parameters, this scenario is most
straightforward to describe as a game rather than
an optimization problem, specifically a mini max game.
Minimax is a strategy of always minimizing
the maximum possible loss which can result from a choice
that a player makes.
And the Nash equilibrium is where
the optimal outcome of a game is one where no player has
an incentive to deviate from his chosen strategy
after considering an opponent's choice.
So the way we train GANs is to find the Nash
equilibrium of a minimax game between G and D.
We'll load our data and initialize both of our models
in our train function.
Then we'll combine them both and initialize
stochastic gradient descent optimizers for both of them.
We'll defined a loss function binary cross entropy, then
perform the set of steps outlined in the paper.
For every time step, we sample data from both distributions,
then update D using our gradients.
Then we update G using our gradients.
Once we're done training, we can generate some images from G.
And we'll see that they do indeed look pretty
similar to our other images.
We can even stitch a bunch of these images
together to make a video.
This can be applied to any image or video data set,
since videos are just collections of images.
We just feed them in one at a time.
All right, so what are the main takeaways here?
Generative adversarial networks are
a framework for generating realistic samples
from random noise.
They consist of two neural nets, a generator that
creates fake samples and a discriminator
that tries to judge if those samples are real or fake.
And we optimize them through back propagation,
which helps find the Nash equilibrium of the minimax game
between both G and D.
First place for last week's coding challenge
goes to Nemanja Tomic.
Nemanja used a convolutional variational auto encoder
to generate videos by training on my own videos.
He also added a recurrent network
to this architecture, which I haven't seen done before.
Very cool results in the Jupiter notebook,
definitely check it out.
Wizard of the week, and the runner up is Niyas Mohammed.
Niyas used a VAE to generate Pokemon
and wrote up a really cool blog post on his whole process.
You guys inspire me.
And I bow to you.
This week's coding challenge is to use
a GAN to generate some video.
Details are in the readme GitHub links.
Go in the comment, and winners are going
to be announced next week.
Please subscribe for more programming videos,
and for now, I've got to discriminate between data.
So thanks for watching.