Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.
When watching science fiction movies, we often encounter crazy devices and technologies that
don’t really exist, or sometimes, ones that are not even possible to make.
For instance, reconstructing sound from vibrations would be an excellent example of that, and
could make a great novel with the secret service trying to catch dangerous criminals.
Except that it has already been done in real life research.
I think you can imagine how surprised I was when I saw this paper in 2014 that showcased
a result where a camera looks at this bag of chips, and from these tiny-tiny vibrations,
it could reconstruct the sounds in the room.
Let’s listen.
Yes, this indeed sounds like science fiction.
But 2014 was a long-long time ago, and since then, we have a selection of powerful learning
algorithms, and the question is, what’s the next idea that sounded completely impossible
a few years ago, which is now possible?
Well, what about looking at silent footage from a speaker and trying to guess what they
were saying?
Checkmark, that sounds absolutely impossible to me, yet, this new technique is able to
produce the entirety of this speech after looking at the video footage of the lip movements.
Let’s listen.
Wow.
So the first question is, of course, what was used as the training data?
It used a dataset with lecture videos and chess commentary from 5 speakers, and make
no mistake, it takes a ton of data from these speakers, about 20 hours from each, but it
uses video that was shot in a natural setting, which is something that we have in abundance
on Youtube and other places on the internet.
Note that the neural network works on the same speakers it was trained on and was able
to learn their gestures and lip movements remarkably well.
However, this is not the first work attempting to do this, so let’s see how it compares
to the competition.
The new one is very close to the true spoken sentence.
Let’s look at another one.
Note that there are gestures, a reasonable amount of head movement and other factors
at play and the algorithm does amazingly well.
Potential applications of this could be video conferencing in zones where we have to be
silent, giving a voice to people with the inability to speak due to aphonia or other
conditions, or, potentially fixing a piece of video footage where parts of the speech
signal are corrupted.
In these cases, the gaps could be filled in with such
a technique.
Look!
Now, let’s have a look under the hood.
If we visualize the activations within this neural network, we see that it found out that
it mainly looks at the mouth of the speaker.
That is, of course, not surprising.
However, what is surprising is that the other regions, for instance, around the forehead
and eyebrows are also important to the attention mechanism.
Perhaps this could mean that it also looks at the gestures of the speaker, and uses that
information for the speech synthesis.
I find this aspect of the work very intriguing and would love to see some additional analysis
on that.
There is so much more in the paper, for instance, I mentioned giving a voice to people with
aphonia, which should not be possible because we are training these neural networks for
a specific speaker, but with an additional speaker embedding step, it is possible to
pair up any speaker with any voice.
This is another amazing work that makes me feel like we are living in a science fiction
world.
I can only imagine what we will be able to do with this technique two more papers down
the line.
If you have any ideas, feel free to speculate in the comments section below.
What a time to be alive!
Thanks for watching and for your generous support, and I'll see you next time!