Hello World, welcome to Sirajology!
In today’s episode we’re going to talk about building an AI Artist.
Art is the embodiment of the human experience, its a synthesis of all of our emotions and
experiences in life.
You might be asking yourself, is it really possible to build something that can do this?
’The Answer is Yes’.
It’s incredible, we can train a neural network to learn an artist’s style, then tell it
to modify a picture into a painting in that style.
It’s using machine learning to repaint an image just like your favorite artist would.
It all started when the Google Research team released a blog post called inceptionism.
They trained a deep convolutional neural network on a huge dataset of images so that it could
start to detect everyday things like dogs and buildings.
Once it was able to do that, they gave it a novel image and asked it to detect an object
it had learned in the image.
if it saw something in the image that looked even slightly similar to what it had already
learned, say a cloud that kinda looked like a dog, it would then modify the image to look
more like a dog.
This resulted in some pretty crazy looking pictures.
Then another group wrote a similar paper called a ‘neural algorithm for artistic style’
where they repainted an image in the style of another, using famous paintings as the
base image.
So when they trained their neural net on starry night then gave it a novel image, it would
modify the image to look more like starry night by artifying all its features.
Since we’ve gone over convolutional neural nets in a previous episode at a high level,
we’re going to really deep dive into the code for this one to try to understand exactly
how this process works.
We’re going to go through the necessary code to recreate the results from the Neural
Style paper in Python using the deep learning library Keras.
Let’s get started.
We’ll start off by defining our arguments.
When we run our script, we define the base image, the style image, and the output image.
We’ll create variables for them, then reference a pre-computed weights file called vgg16.
These are just a bunch of pre-computed synapse weights trained to recognize everyday images
which we’ll later add to our neural net as a starting point.
We’ll also want to initialize booleans that define whether or not to rescale our images
or maintain their aspect ratio based on user input.
Then we initialize our variables for style and content weights.
So what do we mean here by style and content weight?
Well, in the neural style paper they found that when they trained a neural net to recognize
the painting style of an artist, the learned features in the initial layers were style
based.
Like you know how when you train a neural net to recognize a dog, inital layers detect
edges, then the next ones detect shapes, then more complex shapes, then a whole dog?
Well those same layers of abstraction apply to paintings, but what they found were the
initial layers; the edges, and curvatures, and other low level features -- equated with
style.
They also found that the highest few layers were more based on content.
So in the starry night photo the higher levels would be that dope sun thing and perhaps a
collection of stars.
The lower levels would be the curvature of the night sky, and the color schemes.
In this way, CNNs help seperate content from style and mirror the capabilites of our biological
vision.
It really helps you appreciate that the ability to create and enjoy art is a signature of
the inference capabilities of our visual system.
We then need to define how much we want to weigh one or the other because we can optimize
for one of them, or both.
Depending on how much we weigh each, we’ll get a different output.
We’ll set our image dimensions then create a tensor representation of our base image,
style image, and output image.
Then we’ll combine all 3 into a single input tensor.
A tensor is a multi-dimensional array.
So an example would be an array like [colors, textures].
Each of those would be arrays as well so colors would be its own array containing the 7 main
colors.
Then each of those colors would be an array of sub-colors.
We convert our images to a single tensor because its an easily parsable data structure for
our neural network.
Tensors help us reduce the high dimensionality of our images and that in turn reduces the
computational complexity.
Now we we’re going to build our model, which is going to be a convolutional neural network.
we’ll add in our input tensor as the first layer, then we’ll start defining the other
layers.
We’re going to add 31 layers to our neural net.
There are 3 types.
The convolution2D layer type means it has a set of learnable filters which have a small
receptive field.
The receptive field is the subset of filters that are used to connect neurons to a local
region in the next layer, instead of every single other neuron.
The ZeroPadding layer helps us control the size of the output volume by padding zeros
across the border.
Then theres the average pooling layer.
pooling is a concept in CNNs where we take an input image and split it into a set or
pool of rectangles.
Pooling helps us avoid overfitting and reduce the amount of parameters and computation by
only using a subset of the image as a representation rather than all of it.
The idea is that once a feature has been found, the exact location isn’t as important as
its rough location relative to other features.
We'll take the average value from the pool.
The activation function at each of our units is ReLU or rectified linear unit.
The ReLu function is faster than the sigmoid function without a huge difference in generalization
accuracy so we’ll use that.
The numbers here are the number of output filters and the length and width of the input
image.
We’ll name each of our layers as well for reference.
Now that we have our model, we’ll add in the precomptued weights we called earlier.
Then we’re going to define our loss functions for the style, content, and total variation
(aka the uniformity of the image).
Loss functions help us calculate the difference between the expected output and the actual
output.
We’re going to take these loss functions and combine them into a single scalar, or
number.
Then we’ll get the gradients of the generated image using the loss which helps us map the
color scheme.
Our last step is to train our model by minimizing the loss using backpropagation.
The backpropagation algorithm we’ll use in this case is called limited-memory BFGS.
L-BFGS helps minimize our loss function and is space efficient in that it only stores
a few key vectors instead of all of them.
Minimizing the loss function means modifying the output image iteratively so it looks more
and more similar to the artistic style we want.
Scipy gives us this technique as a built in function and After training is over, that
we can rescale and save the output image.
And that’s about it!
This is what happened when I tried out the algorithm.
So beautiful :’( You could also apply this algorithm frame-by-frame to video content,
and with more data and computing power its only gonna get better.
I hope that in the next few years we’ll be able to generate not just paintings but
other forms of art like sculptures, interior design, and facial hair styles.
For more information check out the links down below and please subscribe for more ML videos.
For now I’ve got to go fix a compile time error, so thanks for watching!