Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
Today we are going to talk about video stabilization.
A typical application of this is when we record family memories, and other cool events,
and sometimes, the footage gets so shaky that we barely know what is going on.
In these cases, video stabilization techniques can come to the rescue,
which means that in goes a shaky video, and out comes a smooth video. Well,
that is easier said than done. Despite many years of progress, there is a great
selection of previous methods that can do that, however, they suffer from one of two issues.
Issue number one is cropping. This means that we get usable results,
but we have to pay a great price for it, which is, cropping away a great deal of the video content.
Issue number two is when we get the entirety of the video, no cropping,
however, the price to be paid for this is that we get lots of issues that we call visual artifacts.
Unfortunately, today, when we stabilize, we have to choose our poison.
It’s either cropping, or artifacts. Which one would you choose? That is difficult to decide,
of course, because none of these two tradeoffs are great. So our question today is, can we do better?
Well, the Law of Papers says that of course, just one or two more papers down the line, this will be
way better. So, let’s see…this is what this new method is capable of.
Hold on to your papers, and notice that this will indeed be a full-size video, so we already know
that probably there will be artifacts. But…wait a second! No artifacts. Whoa. How can that be?
What does this new method do that previous techniques didn’t? These magical results are a
combination of several things: one, the new method can estimate the motion of these objects better,
two, it removes blurred images from the videos and three, collects data from neighboring video
frames more effectively. This leads to a greater understanding of the video it is looking at.
Now, of course, not even this technique is perfect - rapid camera motion may lead to warping,
and if you look carefully, you may find some artifacts, usually around the sides of the screen.
So far, we looked at previous methods, and the new method. It seems better. That’s great. But
how do we measure which one is better? Do we just look? An even harder question would be,
if the new method is indeed better, okay, but by how much better is it?
Let’s try to answer all of these questions. We can evaluate these techniques against each other
in three different ways. One, we can look at the footage ourselves. We have already done that,
and we had to tightly hold on to our papers, it has done quite well in this test.
Test number two is a quantitative test. In other words, we can mathematically define how much
distortion there is in an output video, how smooth it is, and more, and compare the output videos
based on these metrics. In many cases, these previous techniques are quite close to each other,
and now, let’s unveil the new method. Whoa. It scored best or second best
on 6 out of 8 tests. This is truly remarkable, especially given that some of these competitors
are from less than a year ago. That is nimble progress in machine learning research. Loving it.
And the third way to test which technique is better, and by how much is by conducting a user
study. The authors have done that too! In this, 46 humans were called in, were shown the shaky
input video, the result of a previous method, and the new method, and were asked three questions.
Which video preserves the most content, which has fewer imperfections, and which is more stable.
And the results were stunning - despite looking at many different competing techniques,
the participants found the new method to be better at the very least 60% of the time,
on all three questions. In some cases, even 90% of the time or higher. Praise the papers!
Now, there is only one question left. If it is so much better than previous techniques,
how much longer does it take to run? With one exception, these previous methods take
from half a second to about 7.5 seconds per frame, and this new one asks for 9.5 seconds per frame.
And in return, it creates these absolutely amazing results. So, from this glorious day on, fewer,
or maybe no important memories will be lost due to camera shaking. What a time to be alive!
Thanks for watching and for your generous support, and I'll see you next time!