(light tinkling music)
- [Announcer] Innovation hinges on our ability
to see things differently.
Breaking boundaries and looking
between the lines in an effort to solve
some of the world's toughest challenges.
(light tinkling music)
- Hi, everybody.
If you've been watching Microsoft Build
you have probably seen one of these innovation videos
happening in this stage.
Well, I know that you know it but this stage is not real.
I'm actually confined at home like the rest of the speakers
in these innovation sessions.
Now the magic behind this is artificial intelligence.
In this video, I want to show you how it works
and how you can use it for your own presentations.
To do that, let me switch to the real video
that I'm recording today.
There you go.
So you can see that I'm actually at home,
welcome to my house.
This is a regular room.
I have a couch, I have furniture,
I even have some toys from my daughter which are everywhere.
Now the magic behind this is the setup that I have
in front of me.
To show you that setup, let me switch to that camera
that you see there, okay?
Let's go for it.
Now this is the setup.
You can see that I have two cameras in here.
Now I have two but I could use only one if I want to.
So this is to switch between the different angles.
Now you look closer you can see
that these are Azure Kinect cameras.
We do that because with Azure Kinect,
I can capture also the depth information.
And that, as we will see later,
it will make my AI model more accurate.
I also have a couple of ambient lights.
Again, not required, you don't need to use them
if you don't want to.
And this is my laptop
where I have the two Kinects connected.
Now in that laptop, I'm running an app
called Speaker Recorder that you can download
from the GitHub Repo.
Let me show you that app.
Now in this app you can see that it's captured
the different video inputs.
I have the RGB signal from the Azure Kinect cameras,
I also have the depth information that we were mentioning
before, and optionally I can also record a PowerPoint video.
Now once I'm happy with the result,
I need to apply the AI model on top of these video inputs.
To do that I can also download the Command Line tool
that is available in the same GitHub Repo.
Now the output of this Command Line tool
is a video like this.
So you can see that it's a transparency mask
with my silhouette on it.
Now I can use this transparency mask
and apply to my original video
so I can remove the background with anything that I want.
For example, the Virtual Stage.
Now I know that if you are a developer,
you want to know all the details.
So let me share some of those details to you.
This AI model is based on the work
from the University of Washington.
They recently published a paper where their researchers
develop an AI model that is doing that magic.
Now in that AI model, that AI model
is taking two video inputs.
The first one, the first image, is the background.
So I need to record the background without the person on it
for the AI model to work.
The second one is the real video,
the one with the person speaking on it.
Now the output of that video is the transparency mask
that you saw before.
But it has some issues.
So for example, if I'm wearing something
that is the color is similar to the background,
I may have gaps or holes in my body.
Not great.
So what they did is using a second neural network.
In this case, this neural network takes only one video input
and it tries to guess the silhouette of the presenter
just by looking at the image.
Because we know that we have a person in there.
Now that output is great but it's not as accurate
as the previous one.
It works great with the background,
but for example, for things like the hair
or the space between fingers, it's not as smooth.
So what they did is combining these two models
with a neural net architecture called context switching.
What it does is picking the neural network
depending on the conditions.
So it takes the best of both worlds.
Now in our case, we could go one step farther,
'cause we have Azure Kinect.
So what we did is replacing the second neural network
with the silhouette that is guessed by the Azure Kinect,
which is much more accurate because it has that depth
information that we were mentioning before.
Now we can improve these even farther.
We can take this model and use another AI technique
called adversarial network.
What they did was connecting the output
to another neural network which is able to identify
if a video is fake or real.
Now if you change the input, if you change the model
slightly, they original model slightly,
and you try to fool this third neural network,
the result will be a neural network, a model,
that is more accurate
and is able to create more natural images.
You have all of these process available in the GitHub Repo,
so we welcome your contributions to make it even better,
and of course it would be great if you used this solution
also to record your own presentations
so we can keep enjoying those community events
while at the same time keeping the speakers safe.
Thank you very much.
(light tinkling music)
- [Announcer] Innovation hinges on our ability
to see things differently.
Breaking boundaries and looking
between the lines in an effort to solve
some of the world's toughest challenges.