Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
First, let's talk about the raster and vector graphics.
What do these terms mean exactly?
A raster image is a grid made up of pixels, and for each of these pixels, and for each
of the pixels, we specify a color.
That's all there is in an image - it is nothing but a collection of pixels.
All photographs on your phone, and generally most images you encounter are raster images.
It is easy to see that the quality of such images greatly depends on the resolution of
this grid - of course, the more grid points, the finer the grid is, the more details we
can see.
However, in return, if we disregard compression techniques, the file size grows proportionally
to the number of pixels, and if we zoom in too close, we shall witness these classic
staircase effects that we like to call aliasing.
However, if we are designing a website, or a logo for a company, which should look sharp
on all possible devices and zoom levels, vector graphics is a useful alternative.
Vector images are inherently different from raster images, as the base elements of the
image are not pixels, but vectors and control points.
The difference is like storing the shape of a circle on a lot of pixels point by point,
which would be a raster image, or just saying that I want a circle on these coorindates
with a given radius.
And as you can see in this example, the point of this is to have razor sharp images at higher
zoom levels as well.
Unless we go too crazy with fine details, file sizes are also often remarkably small
for vector images, because we're not storing the colors of millions of pixels.
We are only storing shapes.
If we want to sound a bit more journalistic we can kind of say that vector images have
infinite resolution.
We can zoom in as much as we wish, and we won't lose any detail during this process.
Vectorization is the process where we try to convert a raster image to a vector image.
Some also like to call this process image tracing.
The immediate question arises - why are we not using vector graphics everywhere?
Well, one, the smoother the color transitions and the more detail we have in our images,
the quicker the advantage of vectorization evaporates.
And two, also note that this procedure is not trivial and we are also often at the mercy
of the vectorization algorithm in terms of output quality.
It is often unclear in advance whether it will work well on a given input.
So now we know everything we need to know to be able to understand and appreciate this
amazing piece of work.
The input is a rough sketch, that is a raster image, and the output is a simplified, cleaned-up
and vectorized version of it.
We're not only doing vectorization, but simplification as well.
This is a game changer, because this way, we can lean on the additional knowledge that
these input raster images are sketches, hand-drawn images, therefore there is a lot of extra
fluff in them that would be undesirable to retain in the vectorized output, therefore
the name, sketch simplification.
In each of these cases, it is absolute insanity how well it works.
Just look at these results!
The next question is obviously, how does this wizardry happen?
It happens by using a classic deep learning technique, a convolutional neural network,
of course, that was trained on a large number of input and output pairs.
However, this is no ordinary convolutional neural network!
This particular variant differs from the standard well-known architecture as it is augmented
with a series of upsampling convolution steps.
Intuitively, the algorithm learns a sparse and concise representation of these input
sketches, this means that it focuses on the most defining features and throws away all
the unneeded fluff.
And the upsampling convolution steps make it able to not only understand, but synthesize
new, simplified, and high-resolution images that we can easily vectorize using standard
algorithms.
It is fully automatic and requires no user intervention.
In case you are scratching your head about these convolutions, we have had plenty of
discussions about this peculiar term before, I have linked the appropriate episodes in
the video description box.
I think you'll find them a lot of fun - in one of them, I pulled out a guitar and added
reverberation to it using convolution.
It is clear that there is a ton of untapped potential in using different convolution variations
in deep neural networks.
We've seen in a DeepMind paper earlier that used dilated convolutions for state of the
art speech synthesis, that is a novel convolution variant and this piece of work is no exception
either.
There is also a cool online demo of this technique that anyone can try.
Make sure to post your results in the comments section!
We'd love to have a look at your findings.
Also, have a look at this Two Minute Papers fan art.
A nice little logo one of our kind Fellow Scholars sent in.
It's really great you see that you've taken your time to help out the series, that's very
kind of you.
Thank you!
Thanks for watching, and for your generous support, and I'll see you next time!