Hey everyone, it’s Frenoy. This is going to be another video on image processing and
today, we are going to learn the principles behind motion detection. I think it is important
for us to understand the concepts before we move onto things like machine learning, object
detection and so on in future videos. This is what the final outcome is going to look
like.
This is the second video on image processing, so be sure to watch the first one where we
learn about and install OpenCV. These videos will also be added to a separate playlist
for your convenience. So let’s dive in.
Firstly, what is motion detection at the fundamental level? Say you have a camera looking at a
scene like this. To the camera, this image is nothing but a series of numbers. Now, when
somebody walks into the frame, there will be a change in the visual output and there
will also be a change in the corresponding numbers that the camera sees. If we can detect
this change, then we can say that we are capable of detecting motion. In summary, whenever
there is motion in a video or image frame, there will be a change in the pixel values.
The next question is, how do you detect the change in pixel values?
For that, I want you to take a look at these two images. They are both identical with the
exception of this object which is a network clock that we built in a previous video. If
you compare the two images, then the pixel values for this region will be different in
both the images. And we will use that to detect the clock using OpenCV. If you think that
subtracting the images will enable you to detect the object, then you are right and
that is what we are going to be doing. But there are a couple of things we need to do
in order to simplify the processing before we can simply subtract the images. Here’s
what the final outcome will look like and this may seem complicated but it is actually
pretty simple and interesting at the same time. We will skim through the process here
and we will take a look at the script towards the end of this video.
In the last video, we learnt that images are made up of red, green and blue components.
In this particular case, we do not need this information so we can simply start by converting
the images into grayscale to give us something like this. This will reduce the number set
or matrices from 3 to just one which will speed up the whole process. If you look closely,
then you will notice that the colour information has been lost.
Since we are only interested in detecting the object, we do not really care about the
edges. Because of this, we can make the processing a bit easier by blurring the image to give
us something like this.
We are now ready to subtract the images and doing that, gives us this result. You can
see the clock in the result along with some traces of other objects. This is primarily
due to changes in the lighting, shadows and so on. We then convert this greyscale image
into a black and white image to give us something like this. The clock is now very prominent
but it is broken up into smaller sections, again due to lighting and so on. This can
be further improved by dilating the image so that nearby pixels are grouped together
to give us this result. And finally, all we need to do is detect the largest blob on the
screen, which represents our clock.
Here’s an overview of what we just did. We obtained two images, image 1 was the reference
with no object, and image 2 contained the object. We then converted them to greyscale,
added some filtering to blur them a bit, subtracted them, converted them to binary or black and
white images, dilated the result to combine some of the nearby pixels or blobs and finally,
we detected the largest blob and drew a rectangle around it.
That is one basic method of detecting changes in pixel values of an image and this is the
principle behind motion or object detection. You can obviously tweak things a bit more
to refine the final output to suit the image conditions and so on.
Now that we know what is going on at the basic level, we can apply this to videos which are
nothing but a series of images. We would need to create a video feed, capture the first
frame as the reference image similar to image 1, and then every other frame will be compared
to this reference frame. The rest of the process is exactly the same as before. Here’s the
script in action. We have the video feed and the output after the frames have been subtracted,
converted to binary and dilated.
Now, if I place my hand in the frame, it will detect it and draw a rectangle around it.
All these white spots indicate objects, but the script is written to detect the largest
blob as seen in the final output. The script also computes the area of the blob and displays
this on the terminal. You can use this area information to determine if the object is
approaching the camera or moving away from it. Notice how the area changes as I move
my hand closer and then further away from the camera.
This script is effective but it has one major drawback, the reference frame is obtained
at the start of the video feed and it is not updated after that. This means that shadows
or changes in lighting can be falsely assumed to be objects when they are not. One way out
of this situation is to update the reference frame at regular intervals which could work.
The other solution is to average out the consecutive frames and this way, the reference frame will
continuously adapt to changes in lighting, shadows and so on. That is what this final
script does. As you can see, there is no motion or object detected right now, but it will
detect my hand as soon as I place it in the frame. Everything else works just like before.
Keep in mind that because we are averaging the frames, the area will continue to decrease
if I keep my hand still. Ultimately this will reduce to zero and the script will assume
that that is no motion or external object as my hand will become part of the reference
image.
Technically this is still correct as there is no motion in the image. But as soon as
I move my hand, it will get detected again.
It is now time to look at the scripts. The description contains links for you to use
them. Simply copy them over to your Raspberry Pi and then open up the built-in python editor
called Thonny.
First, we open up the basic object detection script. It starts by reading the images and
resizing them to speed up the processing. We use the function imshow to display the
images or intermediate steps and they contain the appropriate titles for reference. The
processing flow is the same as what we described earlier. Once the images are obtained, they
are converted to greyscale, then blurred to smoothen the edges, we then obtain the difference
between the two, convert the difference to binary, dilate it, and finally we find the
contours or blobs. Since there can be noise in the image, due to lighting or shadows,
we calculate the areas for each of the contours and we then obtain and use the contour which
has the largest area. In order to display the object, we draw a rectangle and show the
original image.
We can run the script to obtain an output like this. You can click the stop button to
stop the execution. This script opens up a number of windows, and you can comment out
the ones you do not need by using the hash or pound symbol like so.
Next, we have the motion detection script, which obtains a video feed. Again, the script
follows all the stages we spoke about in the video. We start by initializing the camera,
resizing it and then capturing the consecutive frames. This section checks to see if we have
a reference frame and if we do not, then we use the blurred, grayscale image of the first
frame as reference. Rest of the process is similar to the previous script. If you run
this script, then you should obtain something like this.
Finally, we have the version that uses averaging for the reference. It is very similar to the
previous script, with the major addition being these lines which take care of averaging.
You can adjust this final number which is the averaging weight and which determines
how quickly the frames will be averaged out.
The rest of the script is similar to the previous one. Like before, the contour area is also
displayed on the terminal. If you run this script, then you should obtain something like
this. It will detect changes in the frame and highlight the largest contour. But if
you hold your hand steady for too long, it will get averaged out into the reference image.
This can also be seen as the seen as the area continues to decrease. Your hand will only
be detected again once you move it a bit.
The three scripts have been named appropriately and contain suitable comments. You can also
download the reference images for the first script or you can use your own. I would encourage
you to play around with the filtering values, averaging weight and so on to get a better
understanding of their effects.
Parts of these scripts have been obtained from a website called pyImage search that
focuses on computer vision so please have a look at that website if you want to learn
some more. It contains a lot of useful material but do keep in mind that some of it is pretty
advanced.
I hope this video has given you a better understanding of how image processing works in general.
These are just the basics but pretty much all the algorithms use something similar to
achieve the end results. We will most likely look at a very interesting object detection
system in next Sunday’s video so do subscribe to stay updated, if you haven’t already.
Thank you for watching and I will see you in the next one.