Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today, we are going to explore a paper that improves on the incredible StyleGAN2.
What is that?
StyleGAN2 is a neural network-based learning algorithm that is not only capable of creating
these eye-poppingly detailed images of human beings that don’t even exist, but, it also
improved on its previous version in a number of different ways.
For instance, with the original StyleGAN method, we could exert some artistic control over
these images, however, look, you see how this part of the teeth and eyes are pinned to a
particular location and the algorithm just refuses to let it go, sometimes to the detriment
of its surroundings.
The improved StyleGAN2 method addressed this problem, you can see the results here.
Teeth and eyes are now allowed to float around freely, and perhaps this is the only place
on the internet where we can say that and be happy about it.
It could also mix two images together, and it could do it not only for human faces, but
for cars, buildings, horses, and more.
And get this, this paper was published in December 2019, and since then, it has been
used in a number of absolutely incredible applications and followup works.
Let’s look at three of them.
One, for instance, the first question I usually hear when I talk about an amazing paper like
this is “okay, great, but when do I get to use this”?
And the answer is, right now, because it is implemented in Photoshop in a feature that
is called Neural Filters.
Two, artistic control over these images has improved so much that now, we can pin down
a few intuitive parameters and change them with minimal changes to other parts of the
image.
For instance, it could grow Elon Musk a majestic beard…and, Elon Musk was not the only person
who got an algorithmic beard, I hope you know what’s coming…yes, I got one too!
Let me know in the comments whose beard you liked better!
Three, a nice followup paper that could take a photo of Abraham Lincoln and other historic
figures, and could restore their images as if we were time travelers and took these photos
with a more modern camera.
The best part here was that it leveraged the superb morphing capabilities of StyleGAN2
and took an image of their siblings, a person who has somewhat similar proportions to the
target subject, and morph them into a modern image of this historic figure.
This was brilliant, because restoring images is hard, but with StyleGAN2, morphing is now
easy, so the authors decided to trade a difficult problem for an easier one.
And the results speak for themselves.
We cannot know for sure if this is what these historic figures really looked like, but for
now, it makes one heck of a thought experiment.
And now, let’s marvel together at these beautiful results with the new method, that
goes by the name, StyleGAN2-ADA.
While we look through these results, all of which were generated with the new method,
here are three things that it does better.
One, if often works just as well as StyleGAN2 but requires ten times fewer images for training.
This means that now, it can create these beautiful images, and this can be done by training a
set of neural networks from less than 10 thousand images at a time.
Whoa.
That is not much at all.
Two, it creates better quality results.
The baseline here is original StyleGAN2, the numbers are subject to minimization and are
a measure of the quality of these images.
As you see from the bolded numbers, the new method not only beats the baseline method
substantially, but it does it across the board.
That is a rare sight indeed.
And three, we can train this method faster, it generates these images faster, and in the
meantime, also consumes less memory, which is usually in short supply on our graphics
cards.
Now, we noted that the new version of the method is called StyleGAN2-ADA.
What is ADA?
This part means Adaptive Discriminator Augmentation.
What does that mean exactly?
This means that the new method endeavors to squeeze as much information out of these training
datasets as it can.
Data augmentation is not new, it has been done for many years now, and essentially this
means that we rotate, colorize, or even corrupt these images during the training process.
The key here is that with this, we are artificially increasing the number of training samples
the neural network sees.
The difference here is that they used a greater set of augmentations, and the adaptive part
means that these augmentations are tailored more to the dataset at hand.
And now comes the best part, hold on to your papers, and let’s look at the timeline here.
StyleGAN2 appeared in December 2019, and StyleGAN2-ADA, this method came out just half a year later.
Such immense progress, in just 6 months of time.
The pace of progress in machine learning research is absolutely stunning these days.
Imagine what we will be able to do with these techniques just a couple more years down the
line.
What a time to be alive!
But this paper also teaches a very important lesson to us that I would like to show you.
Have a look at this table that shows the energy expenditure for this project for transparency,
but it also tells us the number of experiments that were required to finish such an amazing
paper.
And that is more than 3300 experiments.
255 of which were wasted due to technical problems.
In the foreword of my PhD thesis, I wrote the following: “Research is the study of
failure.
More precisely, research is the study of obtaining new knowledge through failure.
A bad researcher fails 100% of the time, while a good one fails only 99% of the time.
Hence, what you see written here (and in most papers) is only 1% of the work that has been
done.
I would like to thank Felícia, my wife, for providing motivation, shielding me from distractions,
and bringing sunshine to my life to endure through many of these failures.”
This paper is a great testament to show how difficult the life of a researcher is.
How many people give up their dreams, after being rejected once, or maybe two times?
Ten times?
Most people give up after 10 tries.
And just imagine having a 1000 failed experiments and still not even being close to publishing
a paper yet.
And, with a little more effort, this amazing work came out of it.
Failing doesn’t mean losing.
Not in the slightest.
Huge congratulations to the authors for their endurance, and for this amazing work, and
I think this also shows that these Weights and Biases experiment tracking tools are invaluable,
because it is next to impossible to remember what went wrong with each of them, and what
should
be fixed.
Thanks for watching and for your generous support, and I'll see you next time!