Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér.
Pose estimation is an interesting area of research where we typically have a few images
or video footage of humans, and we try to automatically extract the pose this person
was taking.
In short, the input is mostly a 2D image, and the output is typically a skeleton of
the person.
Applications of pose estimation include automatic creation of assets for computer games and
digital media, analyzing and coaching the techniques of athletes, or helping computers
understand what they see for the betterment of robotics and machine learning techniques.
And this is just a taste, the list was by no means exhaustive.
Beyond the obvious challenge of trying to reconstruct 3D information from a simple 2D
image, this problem is fraught with difficulties as one has to be able to overcome the ambiguity
of lighting, occlusions, and clothing covering the body.
A tough problem, no question about that.
An ideal technique would do this automatically without any user intervention, which sounds
like wishful thinking.
Or does it?
In this paper, a previously proposed convolutional neural network is used to predict the position
of the individual joints, and curiously, it turns out that we can create a faithful representation
of the 3D human body from that by means of optimization.
We have had a previous episode on mathematical optimization, you know the drill, the link
is available in the video description box.
What is remarkable here is that not only the pose, but the body type is also inferred,
therefore the output of the process is not just a skeleton, but full 3D geometry.
It is coarse geometry, so don't expect a ton of details, but it's 3D geometry, more than
what most other competing techniques can offer.
To ease the computational burden of this problem, in this optimization formulation, healthy
constraints are assumed that apply to the human body, such as avoiding unnatural knee
and elbow bends, and self-intersections.
If we use these constraints, the space in which we have to look for possible solutions
shrinks considerably.
The results show that this algorithm outperforms several other state of the art techniques
by a significant margin.
It is an auspicious opportunity to preserve and recreate a lot of historic events in digital
form, maybe even use them in computer games, and I am sure that artists will make great
use of such techniques.
Really well done, the paper is extremely well written, the mathematics and the optimization
formulations are beautiful, it was such a joy to read.
Regarding the future, I am pretty sure we're soon going to see some pose and skeleton transfer
applications via machine learning.
The input would be a real-world video with a person doing something, and we could essentially
edit the video and bend these characters to our will.
There are some exploratory works in this area already, the Disney guys for instance are
doing quite well.
There will be lots of fun to be had indeed!
Also, make sure to check out the YouTube channel of Welch Labs, who has a great introductory
series for neural networks, which is in my opinion, second to none.
He also has a new series called "Learning to see", where he codes up a machine learning
technique for a computer vision application.
It is about counting the number of fingers on an image.
Really cool, right?
The quality of these videos is through the roof, the link for both of these series are
available in the description box, make sure to check them out!
Thanks for watching, and for your generous support, and I'll see you next time!