Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
In 2017, so more than 300 episodes ago, we talked about an algorithm that took a 3D model
of a complex object, and would give us an easy to follow, step by step breakdown on
how to draw it.
Automated drawing tutorials, if you will!
This was a handcrafted algorithm that used graph theory to break these 3D objects into
smaller, easier to manage pieces, and since then, learning algorithms have improved so
much that we started looking more and more to the opposite direction.
And that opposite direction would be giving a crude drawing to the machine, and getting
a photorealistic image.
Now that sounds like science fiction, until we realize that scientists at NVIDIA already
had an amazing algorithm for this around 1.5 years ago.
In that work, the input was a labeling which we can draw ourselves, and the output is a
hopefully photorealistic landscape image that adheres to these labels.
I love how first, only the silhouette of the rock is drawn, so we have this hollow thing
on the right that is not very realistic, and then, it is now filled in with the bucket
tool, and, there you go.
And next thing you know, you have an amazing-looking landscape image.
It was capable of much, much more, but what it couldn’t do is synthesize human faces
this way.
And believe it or not, this is what today’s technique is able to do.
Look!
In goes our crude sketch as a guide image, and out comes a nearly photorealistic human
face that matches it.
Interestingly, before we draw the hair itself, it gives us something as a starting point,
but if we choose to, we can also change the hair shape and the outputs will follow our
drawing really well.
But it goes much further than this as it boasts a few additional appealing features.
For instance, it not only refines the output as we change our drawing, but since one crude
input can be mapped to many-many possible people, these output images can also be further
art-directed with these sliders.
According to the included user study, journeymen users mainly appreciated the variety they
can achieve with this algorithm, if you look here, you can get a taste of that, while professionals
were more excited about the controllability aspect of this method.
That was showcased with the footage with the sliders.
Another really cool thing that it can do is called face copy-paste, where we don’t even
need to draw anything, and just take a few aspects of human faces that we would like
to combine, and there you go.
Absolutely amazing.
This work is not without failure cases, however.
You have probably noticed, but the AI is not explicitly instructed to match the eye colors,
where some asymmetry may arise in the output.
I am sure this will be improved just one more paper down the line, and I am really curious
where digital artists will take these techniques in the near future.
The objective is always to get out of the way, and help the artist spend more time bringing
their artistic vision to life, and spend less time on the execution.
This is exactly what these techniques can help with.
What a
time
to be alive!
Thanks for watching and for your generous support, and I'll see you next time!