Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
Let's have a look at some recent results on neural style transfer.
You know the drill, we take a photo with some content, and for example, a painting with
the desired style, and the output is an image where this style is applied to our content.
If this is done well and with good taste, it really looks like magic.
However, for pretty much all of the previous techniques, there are always some mysterious
styles that result in failure cases.
And the reason for this is the fact that these techniques are trained on a set of style images,
and if they face a style that is wildly different from these training images, the results won't
be very usable.
This new algorithm is also based on neural networks, it doesn't need to be trained on
these style images, but it can perform high-quality style transfer, and it works on arbitrary
styles.
This sounds a bit like black magic.
So how does this happen exactly?
First, an autoencoder is trained for image reconstruction.
An autoencoder is a neural network where the input and output image is supposed to be the
same thing.
So far, this doesn't make any sense, because all the neural network does is copy and pasting
the inputs to the outputs.
Not very useful.
However, if we reduce the number of neurons in one of the middle layers to very very few
neurons compared to the others, we get a bottleneck.
This bottleneck essentially hamstrings the neural network, and forces it to first come
up with a highly compressed representation of an image, this is the encoder network,
and then reconstruct the full image from this compressed representation.
This is called the decoder network.
So encoding is compression, decoding is decompression or more intuitively, reconstruction.
This compressed representation can be thought of as the essence of the image which is a
very concise representation, but carefully crafted such that a full reconstruction of
the image can take place based on it.
Autoencoders are previous work, and if you would like to hear more about them, check
the video description as we have dedicated an earlier episode to it.
And now, the value proposition of this work comes from the fact that, we don't just use
the autoencoder as-is, but rip this network in half, and use the encoder part on both
the input style and content images.
This way, the concept of style transfer is much much simpler in this compressed representation.
In the end, we're not stuck with this compressed result, because if you remember, we also have
a decoder, which is the second part of the neural network that performs a reconstruction
of an image from this compressed essence.
As a result, we don't have to train this neural network on the style images, and it will work
with any chosen style.
Hell yeah!
With most style transfer techniques, we are given an output image and we either take it
or leave it because and we can't apply any meaningful edits to it.
A cool corollary of this design decision is that we can also get closer to our artistic
vision by fiddling with parameters.
For instance, the scale and weight of the style transfer can be changed on the fly to
our liking.
As always, the new technique is compared to a bunch of other competing algorithms.
Due to the general and lightweight nature of this method, it seems to perform more consistently
across a set of widely varying input styles.
We can also create some mattes for our target image and apply different artistic styles
to different parts of it.
Local parts of a style can also be transferred.
Remember, the first style transfer technique was amazing, but very limited and took an
hour on a state of the art graphics card in a desktop computer.
This one takes less than a second and works for any style.
Now, as and more new phones contain chips for performing deep learning, we can likely
look forward to a totally amazing future where style transfer can be done in our pockets
and in real time.
What a time it is to be alive!
Thanks for watching and for your generous support, and I'll see you next time!