Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
This name is not getting any easier, is it? It used to be Károly Zsolnai, which was hard
enough, and now this... haha. Anyway, let's get started.
This technique simulates how different objects in a video sound when struck. We have showcased
some marvelous previous techniques that were mostly limited to wooden and plastic materials.
Needless to say, there are links to these episodes in the video description box.
A convolutional neural network takes care of understanding what is seen in the video.
This technique is known to be particularly suited to processing image and video content.
And it works by looking at the silent video directly and trying to understand what is
going on, just like a human would.
We train these networks with input and output pairs - the input is a video of us beating
the hell out of some object with a drumstick.
The joys of research! And the output is the
sound this object emits.
However, the output sound is something that changes in time. It is a sequence, therefore
it cannot be handled by a simple classical neural network. It is learned by a recurrent
neural network that can take care of learning such sequences.
If you haven't heard these terms before, no worries, we have previous episodes on all
of them in the video description box, make sure to check them out!
This piece of work is a nice showcase of combining two quite powerful techniques: the convolutional
neural network tries to understand what happens in the input video, and the recurrent neural
network seals the deal by learning and guessing the correct sound that the objects shown in
the video would emit when struck.
The synthesized outputs were compared to the real world results both mathematically and
by asking humans to try to tell from the two samples which one the real deal is. These
people were fooled by the algorithm around 40% of the time, which I find to be a really
amazing result, considering two things: first, the baseline is not 50%, but 0% because people
don't pick choices at random - we cannot reasonably expect a synthesized sound to fool humans
at any time. Like nice little neural networks, we've been trained to recognize these sounds
all our lives, after all. And second, this is one of the first papers from a machine
learning angle on sound synthesis. Before reading the paper, I expected at most 10 or
20 percent, if that.
The tidal wave of machine learning runs through a number of different scientific fields. Will
deep learning techniques establish supremacy in these areas? Hard to say yet, but what
we know for sure is that great strides are made literally every week. There are so many
works out there, sometimes I don't even know where to start. Good times indeed!
Before we go, some delightful news for you Fellow Scholars! The Scholarly Two Minute
Papers store is now open! There are two different kinds of men's T-shirts available, and a nice
sleek design version that we made for the Fellow Scholar ladies out there! We also have
The Scholarly Mug to get your day started in the most scientific way possible.
We have tested the quality of these products and were really happy with what we got. If
you ordered anything, please provide us feedback on how you liked the quality of the delivery
and the products themselves. If you can send us an image of yourself wearing or using any
of these, we'd love to have a look. Just leave them in the comments section or tweet at us!
If you don't like what you got, within 30 days, you can exchange it or get your product
cost refunded.
Thanks for watching, and for your generous support, and I'll see you next time!