Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Style transfer is an interesting problem in machine learning research where we have two
input images, one for content, and one for style, and the output is our content image
reimagined with this new style.
The cool part is that the content can be a photo straight from our camera, and the style
can be a painting, which leads to super fun, and really good looking results.
We have seen plenty of papers doing variations of style transfer, but I always wonder, can
we can push this concept further?
And the answer is, yes!
For instance, few people know that style transfer can also be done for video!
First, we record a video with our camera, then, take a still image from the video and
apply our artistic style to it.
Then, our style will be applied to the entirety of the video!
The main advantage of this new method, compared to previous ones is that they either take
too long, or we have to run an expensive pre-training step.
With this new one, we can just start drawing and see the output results right away.
But it gets even better.
Due to the interactive nature of this new technique, we can even do this live!
All we need to do is change our input drawing and it transfers the new style to the video
as fast as we can draw!
This way, we can refine our input style for as long as we wish, or until we find the perfect
way to stylize the video.
And there is even more.
If this works interactively, then it has to be able to offer an amazing workflow where
we can capture a video of ourselves live, and mark it up as we go.
Let’s see!
Oh wow, just look at that.
It is great to see that this new method also retains temporal consistency over a long time
frame, which means that even if the marked up keyframe is from a long time ago, it can
still be applied to the video and the outputs will show minimal flickering.
And note that we can not only play with the colors, but with the geometry too!
Look, we can warp the style image, and it will be reflected in the output as well.
I bet there is going to be a followup paper on more elaborate shape modifications as well!
And, this new work improves upon previous methods in even more areas.
For instance, this is a method from just one year ago, and here you see how it struggled
with contour-based styles.
Here is a keyframe of the input video, and here is the style that we wish to apply to
it.
Later, this method from last year seems to lose not only the contours, but a lot of visual
detail is also gone.
So, how did the new method do in this case?
Look!
It not only retains the contours better, but a lot more of the sharp details remain in
the outputs.
Amazing.
Now note that this technique also comes with some limitations.
For instance, there is still some temporal flickering in the outputs, and in some cases,
separating the foreground and the background is challenging.
But really, such incredible progress in just one year!
And I can only imagine what this method will be capable of two more papers down the line.
What a time to be alive!
Make sure to have a look at the paper in the video description, and you will see many additional
details, for instance, how you can just partially fill in some of they keyframes with your style,
and still get an excellent result.
Thanks for watching and for your generous support, and I'll see you next time!