Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
One of the main promises of Virtual Reality, VR in short is enhancing the quality of our
remote interactions.
With VR, we could talk with our colleagues and beloved ones through telepresence applications
that create a virtual avatar of us - much like the ones you see here.
Normally, this requires putting sensors all over our faces to be able to reconstruct the
gestures we make.
A previous work used a depth camera that was hanging off of the VR headset, thus having
a better look of the entirety of our face, while a later work used a mouth camera to
solve this problem.
This new paper attempts to capture all of our gestures by using a headset without these
additional complexities by using no more than three infrared cameras.
No extra devices hanging off of the headset, nothing.
All of them are built into the headpiece.
This means two key challenges: one is the fact that the sensor below sees the face in
an uncomfortable, oblique angle, below you see exactly the data that is being captured
by the three sensors.
And two, the output of this process should be a virtual avatar, but it is unclear what
the correspondence between all this data and the animated character should be.
So the idea sounds great, the only problem is that this is near impossible.
So how did the researchers end up doing this?
Well, what they did is they built a prototype headset with 6 additional sensors.
Now, wearing this headset would perhaps not be too much more convenient than the previous
works we’ve looked at a moment ago.
But don’t judge this work just yet, because this additional information is required to
create the output avatar, and then, the smaller three-sensor headset can be trained by dropping
these additional views.
In short, the augmented, more complex camera is used as a crutch to train the smaller headset.
Amazing idea, I love it.
Our more experienced Fellow Scholars also know that there is a little style transfer
magic being done here, and finally, all of these partial views are then stitched together
into the final avatar.
You can also see here that it smokes the competition, uses only three sensors, and does all this
in real time.
Wow.
What an amazing paper.
If you want to show your friends how you are about to sneeze in the highest possible quality
video footage, look no further.
Now, I am a research scientist by day, and I also run my own projects where I cannot
choose my own hosting provider, and every time I have problems with it, I tell my wife
that I wish we could use Linode.
Linode is the world’s largest independent cloud hosting and computing provider, and
they just introduced a GPU server pilot program.
These GPU instances are tailor-made for AI, scientific computing and computer graphics
projects.
Yes, exactly the kind of works you see here in this series.
If you feel inspired by these works and you wish to run your own experiments or deploy
your already existing works through a simple and reliable hosting service, make sure to
join over 800,000 other happy customers and choose Linode.
Note that this is a pilot program with limited availability.
To reserve your GPU instance at a discounted rate, make sure to visit https://www.linode.com/papers
or click the link in the description and use the promo code “PAPERS20” to get 20 dollars
free on your account.
You also get super fast storage and proper support if you have any questions.
Give it a try today!
Our thanks to Linode for supporting the series and helping us make better videos for you.
Thanks for watching and for your generous support, and I'll see you next time!