Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
I got so excited by the amazing results of this paper. I will try my best to explain why, and by
the end of this video, there will be a comparison that blew me away and I hope you will appreciate
it too. With the rise of neural network-based learning algorithms, we are living the advent
of image generation techniques. What you see here is a set of breathtaking results created
with a technique called StyleGAN2. This can generate images of humans, cars, cats, and more.
As you see, the progress in machine learning-based image generation is just stunning.
And don’t worry for a second about the progress in text processing, because that is also
similarly amazing these days. A few months ago, OpenAI published their GPT-3 model that they
unleashed to read the internet, and learn not just our language, but much, much more. For instance,
the internet also contains a lot of computer code, so it learned to generate website layouts from a
written description. But that’s not all, not even close, to the joy of technical PhD students around
the world, it can properly typeset mathematical equations from a plain English description as
well. And get this, it can also translate a complex legal text into plain language,
or, the other way around. And it does many of these things nearly as well as humans.
So what was the key to this work? One of the keys of GPT-3
was that it uses a neural network architecture that is called the transformer network. These
really took the world by storm in the last few years, so our first question is, why transformers?
One, transformer networks can typically learn on stupendously large datasets, like the whole
internet, and extract a lot of information from it. That is a very good thing. And two,
transformers are attention-based neural networks, which means that they are good at learning and
generating long sequences of data. Okay, but how do we benefit from this? Well, when we ask
OpenAI’s GPT-3 to continue our sentences, it is able to look back at what we have written
previously. And it looks at not just a couple of characters, no-no, it looks at up to several
pages of writing backwards to make sure that it continues what we write the best way it can.
This sounds amazing. But what is the lesson here? Just use transformers for everything and off we
go? Well, not quite. They are indeed good at a lot of things when it comes to text processing tasks,
but they don’t excel at generating high-resolution images at all. Can this be improved somehow?
Well, this is what this new technique does, and much, much more.
So let’s dive in and see what it can do! First,
we can give it an incomplete image and ask it to finish it.
Not bad… but! OpenAI’s Image-GPT could do that too, so what else can it do? Oh boy, a lot more!
And by the way, we will compare the results of this technique against Image-GPT at the end of
this video, make sure not to miss that, I almost fell off the chair, you will see in a moment why.
Two, it can do one of my favorites, depth to image generation. We give it a depth map,
which is very easy to produce, and it creates a photorealistic image that corresponds to it,
which is very hard. We do the easy part, the AI does the hard part. Great!
And with this, we not only get a selection of these images, but since we have their depth maps,
we can also rotate them around as if they were 3D objects. Nice!
Three, we can also give it a map of labels, which is, again, very easy to do,
we just say here goes the sea, put some mountains here, and the sky here,
and it will create a beautiful landscape image that corresponds to that.
I can’t wait to see what these amazing artists all over the world will be able to get out of these
techniques, and these results are already breathtaking…but research is a process,
and just imagine how good they will become two more papers down the line. My goodness!
Four, it can also perform super resolution. This is the CSI thing where in goes a blurry
image, and out comes a finer, more detailed version of it. Witchcraft.
And finally, five, we can give it a pose, and it generates humans that take these poses.
Now, the important thing here is that it can supercharge
transformer networks to do these things at the same time, with just one technique.
So how does it compare to OpenAI’s Image completion technique? Well, remember,
that technique was beyond amazing, and set a really high bar. So let’s have a look together!
They were both given the upper half of this image, and had to fill in the lower half. Remember,
as we just learned transformers are not great at high-resolution image synthesis.
So here, for OpenAI Image-GPT we expect heavily pixelated images….and…oh yes, that’s right.
So now, hold on to your papers, and let’s see how much more detailed the new technique is.
Holy mother of papers! Do you see what I see here? Image-GPT came out just a few months ago,
and there is already this kind of progress. So there we go, just imagine what we will be able
to do with these supercharged transformers just two more papers down the line.
Wow. And that’s where I almost fell off the chair when reading this paper. Hope you held
on to yours. It truly feels like we are living in a science fiction world. What a time to be alive!
Thanks for watching and for your generous support, and I'll see you next time!