Hello world
It's Suraj and just a few days ago Facebook's AI research team released a demo
Called dense pose
They were able to map out every single pixel of a human body in a given video and not just for a single human
But for many many humans all at once even more impressive
They were able to do this in videos with lots of distractions and general chaos on a single GPU
Them nope just deep learning. This is pretty cool, and has tons of applications, and we're gonna learn how they did it in this video
Creating 3d animated characters is a lot of work the tools to do so are expensive and require
teams of people working for months on end
Big production studios with the available budgets are able to track humans and convert them to animations
but for people without big budgets this technology will
Democratize that ability remember my video on the deep fakes algorithm
Well this can be used for a full-body version of that if we can track a human
We can swap out their entire body with a different one lending to some surreal scenarios like geoff hinton
charging into Mordor instead of Aragorn augmented reality
Applications could use this to label people that you see in real time with all sorts of statistics
And it would give away four machines to be able to better read our body language for everything from
Sentiment analysis to full-body tracking to virtual reality gaming. This is the new state of the art
Inhuman pose estimation and to understand how it works we're gonna have to try and work through the thought process
The authors had both in terms of building on previous discoveries and trying different methods
So let's say we have a video of a human dancing. This is a
2-dimensional grid of pixels, but when you or I look at it
We're able to tell that there are indeed 3d objects that are being represented by this
2d grid we want our computer to have this ability as well and in a way that
Can visualize it meaning will want to transform this 2d human into a 3d model once?
We have that 3d model we can do whatever we want with it put it into any sort of scene
Swap it with another 3d model change its features lots of possibilities, but all of these possibilities
Depend on us being able to construct a 3d
model of this human in real time that updates its movements as the human moves we want to construct a
Correspondence this is a computer vision term that is a measure of how well pixels in one image
correspond to pixels in another image in our case
It would be a 2d to 3d image, and we don't want any holes in our image
We want all the pixels to be as close together as possible, so we'll call it a dense
Correspondence and we'll just focus on doing this for humans for now to create a dense correspondence
We'll need to perform some object detection
Object segmentation and pose estimation, that's a multi-level problem, so we want to start with a method
That's as simple as possible. We've got so many different avenues to go about this but
Ideally we could use some sort of labeled data set because having a label just makes training
machine learning algorithms much easier since it just involves a learning a function that
Represents the mapping between the input data and the labels in this case
What would the ideal label datasets look like there are tons of labeled image datasets out there these days
Pascal see far image net these are basically big collections of images of a
variety of random objects each with their own
Particular label these data sets have been a catalyst for all the recent progress in computer vision algorithms
But we don't want to just label a human if we see one in a video
We want to be able to create a 3d model of one
There's not really a data set of human images with the label being the 3d model
So we'd have to create one ourselves
And it would involve humans manually creating annotations that relates 3d images to surface based
representations of the human body and that's exactly
What did they basically asked the annotators to annotate regions?
Corresponding to defined body parts like the head and the legs all of these annotations were labeled with their corresponding
3d body parts which acted as the label. They did this for 50k humans, which summed up to be a total of five million
manually annotated
Correspondences once we have our data set we're gonna have to decide on what model to build
We know that deep learning is the state of the art when it comes to classification and that?
Convolutional networks are the state of the art in image classification
but it's not just
Classification we're trying to do, but regression as well
What's the next move the human is going to make the next question would be has anyone done something similar with a?
convolutional network of course
Luckily there was a recent
Architecture called dense reg that did this for all sorts of objects and got decent results so we could start with that
architecture in the first step the network will classify a pixel as
Belonging to either the background or one of several region parts that gives a rough estimate of surface coordinates this is
Essentially a labeling task that can be trained using gradient descent in the second step a regression model
will indicate the exact coordinate of the pixel within the region part a
More formal way of saying this is that in the first stage it will assign position L to the body Part C
that has the highest likelihood as
calculated by the classification
Branch and in the second stage it will use the regressor to place the point L in the continuous coordinate pair
UV that is a
Parameterization of Part C C can take 25 values one would be the background meaning that P is a 25 Way
classification unit and we can train 24 regression functions are
Each of which provides 2d coordinates within its respective parts both the classification and regression
Tasks are trained by minimizing a respective loss function
But the regression loss is only taken into account for a part if the pixel is within
Specific part so our network can work
But it requires a lot of tasks for a single Network like part segmentation and pixel localization. They use a technique called
Region of interest pooling to create different regions and fed the results seeing features into region specific branches
This decomposed the complexity of the task
Into controllable modules all of which which could be trained jointly in an end-to-end approach
So it's a fully convolutional network on top of region of interest pooling that is entirely devoted to two tasks
generating a classification and regression
Head that provide part assignment and part coordinate predictions to improve the model they used a technique called
cascading meaning using a collection of models using all the information collected from the output of one model as
additional information for the next
classifier in the cascade the output of the region of interest a line module feeds into the dense network as well as
Networks for the masking and key points tasks
the first stage predictions from all tasks are then combined and fed into a second stage refinement unit of each branch when
Experimenting on novel videos their architecture worked well across the board for not just one human, but multiple humans
Unfortunately, they haven't released the initial source code
Just yet
But we can recreate what they did using the details in the paper also the data set they created is open source so definitely check
That out three things to remember from this video. Dense pose is a new deep learning model that can estimate 3d models of
multiple humans from just a video on a single GPU they use a collection of
Convolutional networks to do this performing
classification for assignment and
regression for coordinate predictions and
They open source the data set they created mapping humans to their corresponding 3d models
Please subscribe for more programming video and for now I've got to make a goodbye pose so thanks for watching