hello and welcome to red stapler channel in this video we are going to show you
how to use machine learning and javascript to realtimely process the
video and remove the person from it ready let's check it out
first let's add video tag to our HTML page
I'll set video attributes to mute to remove the sound loop and controls for
the control bar we also need to have the canvas so we can display the output
processed video
the current page should look like this original video on the left and output
canvas on the right
next I'm going to create an initialize function
first we will get our video element which will be used later
next is to get output canvas 2d context when working with canvas we'll work with its context instead
of canvas element itself
then I'm going to create another canvas as a temporary working space
we will use this canvas to capture each video frame and image data then process it and
put it to the output canvas for display
after these setup steps we will start playing the video and process it we will
create another function calls computeFrame
don't forget to declare variables before you continue
the computeFrame function will be executed for each video frame
first we will use the temp canvas and draw the current video frame on it using
draw image method
then get the pixel data on a canvas using get image data method
we will work on the processing later now we will just put the video frame on the
output canvas directly
then use settimeout to call itself recursively and start the rendering loop
then we will execute our code after our page contents was loaded
at this point you should see the same video content playing on the output canvas
next we will work on a machine learning part we will use tensorflow.js
a javascript library for machine learning combined with bodypix which
is a real-time person segmentation pre-trained machine learning model we
will use bodypix to detect person on each video frame and then remove them
first let's install both tensorflow and bodypix using CDN and script tag
before we load the machine learning model we will also need to provide a
configuration so let's define it
first is the neural network architecture you can choose between MobileNet and
ResNet in this tutorial we are going to use MobileNet which is faster but has
less accuracy
setting is outputStride which we will set it to you 16 higher number will
increase the accuracy but reduce the processing speed
next is multiplier again higher number will trade accuracy for speed I was
added to one which is maximum and then quantBytes we will set it to 4 which
means use the default model size then I'm going to load the bodypix model
with our configuration
after the model is loaded we will assign the loaded model to global variable
also moved the init function inside the promise
inside the computerFrame we will call segmentperson method to start analyzing
the image
we will pass temp canvas that has the current video image on it followed by a configuration
the first setting is internal resolution which you can choose to downsize the
input image before processing it to increase the performance in this
tutorial I will set it to full which means no resizing but you can switch to
a high medium and low next is segmentation threshold this is
the minimum confident threshold before each pixel is considered part of human
body I will set it to 0.01 which means except
anything higher than 1% confident the last one is score threshold this is
the minimum confident threshold to recognize the entire human body again I
was added to 0.01 then back to the processing after the
segmentation is completed the method will return the result array so let's
start removing the person
first we will get the current image data from the output canvas
then create a nested loop to iterate through every pixels in the image
since the image data is in single dimension array format I'm going to
create an index variable n to access each pixel
now here is the segmentation result from the model. It's a single dimension
array contained only zero and one any non-human pixels will have zero value
so I'm going to create an if statement
to check if the current pixel is not part of human
then we will copy the pixel data from the video to the output canvas but if
it's human we will skip the updating process and use the previous frame data
this is a simple trick to remove the person's from the video
you will notice that I multiplied index the by 4. This because each pixel has four
datas RGB and alpha which requires four array bases
the final step is to put the result to the output canvas
also move the update inside the promise
and that's it let's see the result
you might have to readjust the model configuration for each video like this
example setting very low threshold will resulted in non-human pixels being
removed so I increased the threshold to 5% and everything looks good
please note that this trick have couple of flaws first if there is always human on
a certain area of the video it won't work
also this technique requires fixed background video so the camera cannot
move zoom or pan
and that's all for this tutorial hope you guys enjoy if you want to see more
dev tips and tutorials subscribe our channel to stay tuned thanks watching
see you next video bye