this new background removal technique can extract a person from a single input image without the
need for a green screen or additional information in real time at up to 63 frames per second
let's see how they achieve that and more awesome results
this is what's ai and i share artificial intelligence news every week
if you are new to the channel and want to stay up to date please consider subscribing to not miss
any further news human matting is an extremely interesting task where the goal is to find any
human in a picture and remove the background from it it is really hard to achieve due to the
complexity of the task having to find the person or people with the perfect contour
many techniques are using basic computer vision algorithms to achieve this task
such as the grab cut algorithm which is extremely fast but not very precise this grab cut algorithm
basically estimates the color distribution of the foreground item and the background using a
gaussian mixture model this is why we often use a green screen helping the algorithm to
remove only the green pixels and leave the rest into the final results but the results
are not so great when we do not have access to such a green screen modern deep learning and the
power of our gpus made it possible to create much powerful applications that are yet not perfect
the best example here is deep image matting made by adobe research in 2017 a version of
this model is currently used in most websites you use to automatically remove the background
from your pictures unfortunately this technique needs two inputs an image and its try map a try
map is basically a representation of the image in three levels the background the foreground
and a region where the pixels are considered as a mixture of foreground and background looking
like this to successfully remove the background using the deep image matting technique we need
a powerful network able to localize the person somewhat accurately then we produce a segmentation
where the pixels equivalent to the person are set to one and the rest of the image is set to zero
then we use basic computer vision transformations to create a tri-map from this segmentation
we start by reducing the size of the segmented object to leave a bit of space for the unknown
region by eroding it removing some pixels at the contour of the object iteratively then we
add this third section which is the unknown region by dilating the object adding pixels
around the contour producing a result like this this tri-map is the one we sent to the deep image
mapping model with the original image and you get your output you can see how much computation power
is needed for this technique using two powerful models if you would like to achieve somewhat
accurate results as you just saw the current state-of-the-art approaches are quite accurate
but they need a few seconds and sometimes up to minutes to find the results for a single image
you can just imagine the time it will need to process the whole video
fortunately for us this new technique can process human matting from a single input image
without the need for a green screen or a tri-map in real time at up to 63 frames per second they
call their network mud net it's a lightweight matting objective decomposition network which
we will further detail they train their network in both a supervised and self-supervised way
the supervised way takes an input and learns to remove the background based on a corresponding
ground truth just like usual networks then there is a self-supervised training process
this is called self-supervised because this network does not have access to the ground truth
of the videos it is trained on it uses unlabeled data and has access to the information found in
the previous step which are the parameters of the network it basically takes what the first network
learned and understands the consistency between the object in each frame to correctly remove the
background these two trainings are made on the modnet architecture magnet is basically composed
of three main branches there is a low resolution branch which estimates the human semantics
then based on these results the original image and the ground truth of the image
a high resolution branch focuses on detecting precise human boundaries
finally a fusion branch also supervised by the whole ground tooth map is added to predict the
final result of the alpha matte which will be used to remove the background of the input image
this network architecture is way faster because it first finds the semantic estimation itself
using a basic decoder inside the low resolution branch making it much faster to process as you
can see the network is basically mainly composed of down sampling convolutions and up sampling
an arbitrary cnn architecture can be used where you see the convolutions happening in this case
they used mobile net v2 because it was made for mobile devices it is a small network and extremely
efficient when compared to other state-of-the-art architectures if you are not familiar with
convolutional neural networks or cnns i invite you to watch the video i made explaining what they are
it will be in the top right of the screen and linked in the description below the dawn sampling
and the use of fewer convolutional layers in the high resolution branch is done to reduce
the computational time this fusion branch is just a cnn module used to combine the semantics
and details where enum sampling has to be made if we want the accurate details around the semantics
finally the results are measured using a loss highly inspired from the deep image matting paper
it calculates the absolute difference between the input image and the composited image obtained
from the ground 2 foreground and the ground truth background now there's one last step to
this network's architecture if we come back to the full architecture here we can see that they apply
what they called a one frame delay which uses the information of the precedent frame
and the following frame to fix the unknown pixels hesitating between foreground and
background here you can see an example where the foreground moves slightly to the left
in three consecutive frames and the pixels does not correspond to what it is supposed to
with the red pixel flickering in the second iteration then you have your final results
with the foreground object extracted which is a person in this case and you can add in
many different backgrounds of course this was just a simple overview of this new paper
i strongly recommend to read the paper for a deeper understanding of this new technique
the code and the pre-trained model will also be available soon on their github as they wrote on
their page both are linked in the description below please leave a like if you went this far
in the video and since there are over 90 of you guys watching that are not subscribed yet consider
subscribing to the channel to not miss any further news clearly explained thank you for watching
you