Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
A few years ago, we have mainly seen neural network-based techniques being used for image
classification.
This means that they were able to recognize objects, for instance, animals and traffic
signs in images.
But today, with the incredible pace of machine learning research, we now have a selection
of neural network-based techniques for not only classifying images, but, also, synthesizing
them!
The images that you see here and throughout this video is generated by one of these learning-based
methods.
But of course, in this series, we are always obsessed with artistic control, or, in other
words, how much of a say we have in the creation of these images.
After all, getting thousands and thousands of images without any overarching theme or
artistic control is hardly useful for anyone.
One way of being able to control the outputs is to use a technique that is capable of image
translation.
What you see here is a work by the name CycleGAN!
It could transform apples into oranges, zebras into horses, and more.
It was called CycleGAN because it introduced a cycle consistency loss function.
This means that if we convert a summer image to a winter image, and then back to a summer
image, we should get the same input image back, or at least, something very similar.
If our learning system obeys this principle, the output quality of the translation is going
to be significantly better.
Today, we are going to study a more advanced image translation technique that takes this
further.
This paper is amazingly good at daytime image translation.
It looks at a selection of landscape images, and then, as you see here, it learns to reimagine
our input photos as if they were taken at different times of the day.
I love how clouds form, and move over time in the synthesized images, and the night sky
with the stars is also truly a sight to behold.
But wait, CycleGAN and many other followup works did image translation, this also does
image translation, so, what’s really new here?
Well, one, this work proposes a novel upsampling scheme that helps creating output images with
lots and lots of detail.
Two, it can also create not just a bunch of images a few hours apart, but it can also
make beautiful timelapse videos, where the transitions are smooth.
Oh my goodness.
I love this.
And three, the training happens by shoveling 20 thousand landscape images into the neural
network, and it becomes able to perform this translation task without labels.
This means that we don’t have to explicitly search for all the daytime images and tell
the learner that these are daytime images, and these other images are not.
This is amazing because the algorithm is able to learn by itself, without labels, but it
is also easier to use because we can feed in lots and lots more training data without
having to label these images correctly.
As a result, we now know that this daytime translation task is used as a testbed to demonstrate
that this method can be reused for other kinds of image translation tasks.
The fact that it can learn on its own and still compete with other works in this area
is truly incredible.
Due to this kind of generality, it can also perform other related tasks, for instance,
it can perform style transfer, or in other words, not just change the time of day, but
reimagine our pictures in the style of famous artists.
I think with this paper, we have a really capable technique on our hands that is getting
closer and closer to the point where they can see use in mainstream software packages
and image editors.
That would be absolutely amazing.
If you have a closer look at the paper, you will see that it tries to minimize 7 things
at the same time.
What a time to be alive!
This episode has been supported by Weights & Biases.
Here, they show you how to build a proper Convolutional Neural Network for image classification,
and how to visualize the performance of your model.
Weights & Biases provides tools to track your experiments in your deep learning projects.
Their system is designed to save you a ton of time and money, and it is actively used
in projects at prestigious labs, such as OpenAI, Toyota Research, GitHub, and more.
And, the best part is that if you are an academic or have an open source project, you can use
their tools for free.
It really is as good as it gets.
Make sure to visit them through wandb.com/papers or just click the link in the video description
and you can get a free demo today.
Our thanks to Weights & Biases for their long-standing support and for helping us make better videos
for you.
Thanks for watching and for your generous support, and I'll see you next time!