hello welcome I'm here in this video to
talk some more about computer vision
computer vision being this idea of the
having a computer see have a computer
find something if you notice there's
this strange red cup this actually isn't
part of my computer screen this is a red
cup here taped to the wall behind me and
in a second I might have a camera
pointed at that and I want the computer
to be able to find that red cup and this
is going to open up a lot of questions
around all sorts of different techniques
and strategies of getting things that
are faster and more accurate in that
sort of thing so the first thing that I
want to do though is reference for you a
source of inspiration for me which is an
article by Golan Levin called computer
vision for artists and designers
pedagogic tools and techniques for
novice programmers so there's a link to
this article in the description it gives
you some historical context and it also
walks you through some sort of key
classic techniques like detecting motion
detecting presence detection through
brightness thresholding simple object
tracking etc and I'm going to
demonstrate a lot of the techniques
which I've learned and taken inspiration
from this particular essay online essay
so ok so the other thing I'll mention
briefly is what my my strategy here for
this this set of tutorials is going to
be a whole bunch of them by the time I'm
done there could be up to 10 even is
that I'm going to show you how to
program some of these classic algorithms
from scratch and then what from there
I'm going to transition to showing you
how to use different computer vision
libraries that have these algorithms and
more sophisticated versions of these
algorithms baked into them so it's a
question of you know in practice a lot
of the time you probably want to just
use a library like OpenCV is one of the
ones I'm going to show you but there's
some value in understanding the
algorithm for when for when you go to
use the library for knowing how it works
plus you might invent some sort of wacky
creative new ideas around computer
vision and just sort of creative
visualization by having by playing
around with these by right of the
algorithms yourself ok so a previous
video I started with an example that
does exactly this so and this is an
example running and processing a Java
based platform I will make p5.js
JavaScript versions of these examples
which you can find it hopefully linked
in this video's description as well but
here you have a cup a cup
to a green wall behind me and what you
can see here is that I'm drawing a
circle at that cups location how did I
do that
the way that this particular algorithm
works is it says look at every single
pixel is that pixel rent is that pixel
really rent is that pixel more red than
the other one I found that was kind of
red it's looking for the most red pixel
for every pixel find its color distance
it's difference from red and find the
one that holds the record for being the
most red and then draw a circle at that
location but there's an issue while this
is working somewhat accurately I could
even take this off the wall you can see
it's kind of following it around but it
is sort of jiggling I could I could also
demonstrate another problem by having
two cups now and like now it's jumping
back and forth between the two so
there's a lot of things that I want to
resolve here in these in the way that
I'm going to improve these examples the
first thing that I want to do is simply
make it so that I can get a stable
location hopefully in the center of this
particular cup and so the first thing I
can do at this particular example and
also by the way notice now it's not
finding it anymore
this camera is doing some like auto
white balancing brightness stuff which
is really can be a problem and I think
there's some ways of turning that off
but in the sort of real world if you
take this to an installation or a
project you're making you're going to
want to have a camera that you can turn
off auto white balance auto all those
sort of other automatic settings but I
can in this example I can always click
and kind of recalibrate the color that
I'm tracking so the first thing I want
to do is instead of looking for in this
particular video instead of looking for
the single-most pixel that is the most
red I want to say is a pixel red enough
find all of those pixels and give me the
average location of all those pixels so
let's dive into the code and make that
change okay
as you'll notice here every single
whoops the core algorithm of essentially
all computer vision I have let me look
over here for a second the core
algorithm of all of these computer
vision examples is this this nested loop
for every X pixel for every Y pixel
and something is really bothering me
here I know why these extra spaces are
here and I want to sort of do Auto for I
need to put some spaces here cuz I have
a problem please don't be like me live
your life with spaces in weird places
okay
but and you can see here what am i doing
I am taking and let's just let's just
review this for a second I have a new
whiteboard marker that I've law there it
is
let's just review this idea really
briefly in case you didn't watch the
previous video this is a grid of pixels
it has columns which I could number like
this 0 1 2 3 4 it has rows which I could
number like this and let's just make
those different for the sake of argument
then it has pixels the pixels are
numbered like this 0 1 2 3 4 5 6 7 8 9
10 11
et cetera every single computer vision
algorithm that I demonstrate to you will
start with for every X for every Y so
it's looking at every X pixel and every
Y pixel but then it needs to look up a
given pixel here in the one dimensional
array of all of the pixel colors and the
formula for doing that is X plus y times
width and you can see why that would
work because here the width is 6 the
first pixel in column 3 and X 3 is 3 the
next one is 3 plus 6 which is 9 the next
one is 9 plus 6 which is 15 is that
right 11 12 13 14 15 so this formula is
going to be kind of key so that's right
here right for every X for every I find
that pixel location and then it's look
and then I need to separate out the red
green and blue components of the color
that's coming from the pixels of that
video and then there's this other color
called track color that's the color that
I'm looking for the red is that here yes
that's the red cup and then using the
distance formula and to be honest I
should really not use the distance
formula because it has a square root I
should use just distance squared because
then I don't have to take the square
root but I'm not going to worry about
that optimization right now maybe I'll
bring that in later I'm
you see if it's the if it beats that
sort of record it's the most read and
have that XY spot so that's what this
code is doing so now in the next version
of this example instead of looking for
instead of looking for the most read I
want to look for anything that meet that
that is like read enough so the first
one thing we need is a threshold
variable so I'm going to add to this
program I'm going to add a variable
called threshold and I'm going to try
this just at like 25 I sort of made that
up 25 I want any given pixel that is
within that that that's difference is
less than 25 from the color that I'm
looking board for to be considered red
so let's go down to the algorithm and I
no longer looking for a record or a
closest X or closest Y what I'm really
looking for is like the average X and
the average Y and I'm going to make
those floats because yeah I think that's
going to help out a little bit just in
terms of doing the average stuff and
then I also need to have a count how
many pixels are red okay so I need to I
need to ultimately what I want to do at
the end is have calculated the average
and then I also wanted to keep track of
how many pixels because that's how I'm
going to calculate the average by the
way okay so now we're looking through we
find the current color we know the track
color we know that difference so I'm no
longer checking and I'm going to save
this by the way is computer vision - I'm
no longer checking is that distance less
than the record I just want to know is
it less than that given threshold if
it's if it is red somewhat red enough
then what do I do
average X I add its X location in right
I add the Y location in and then I
increase the count
y now those values aren't really
averaged yet yet how do you take the
average of five numbers you add all five
up and you divide by five so this is
what ultimately I'm going to do here
over here I'm going to add up all the
X's and Y's keep track of how many X's
Y's I add it up and then take the
average so now at the end I can say as
long as count let's just say as long as
we found at least one pixel you know I
could say like you know what this only
works if I find five pixels that might
be more accurate but I'm just say as
long as count is greater than zero I'm
going to say average x equals average X
divided by count average y equals
average Y divided by count I'm going to
fill with the track color give it a
stroke and then draw the circle at
average X average Y so you can see now
here instead of looking for the
single-most pixel any pixel within that
particular threshold of red is is going
to be considered right so let's run this
now and see what we get now I don't know
if I picked a very good value with value
25 and I should really make that a
variable that I can tweak in real time
but let's just sort of see okay now what
I'm going to do is click here to set
that color you can see that's pretty
good it's it actually is still sort of
like moving around one thing that I
think could be useful to do is I could
actually show you which pixels I found
and what I'm going to do to do that I
think is just do I'm going to this is
like not so great but I'm going to draw
just like a point there to sort of see a
white point to see what it's sort of
finding and what the other thing I want
to do is I'm going to just right now I'm
going to map the the threshold between
Mouse X like the lowest threshold would
be zero the mouse x goes between zero
and width and I could have a threshold
between zero and like let's just save
100 so this might make things run kind
of slow because I'm going to be using
the point function a lot but let's sort
of see what happens here okay so let me
click here and see oh why are those all
so first of all I did I didn't I say
photos by the way is a stroke and I want
to say stroke weight one and actually
just for a set
in here let's take let's like like let's
do this and let's make this a little bit
smaller so we can really see what's
going on here okay so let's run this and
we should see white a white pixel for
every pixel it's finding that is red
like and you can see what it's found
here now oh look at this so the
threshold is very high so you can see
even my hand is kind of within the
threshold now so and you can see as I
move the mouse that threshold you know
my skin is kind of like reddish but as I
move over here you can kind of see see
what's going on now I want it Wow my
lips I got to get out of the frame so
you can see now it's quite stable there
you go so I found kind of an appropriate
threshold I should be I have a way of
like printing that value out so I know
and you can see that it's UPS as soon as
I move in the picture we're going to
we're going to deal with that in a
second I want to zoom in here I should
be able to unfortunately my arm is too
red I should be able to take this off
and move this around and you can see
that I have quite a pretty accurate
tracking as I move this and it's it's
quite stable so you can see that even
just doing the average location you know
again I have nice lighting in here I
have a green background I have a very
distinct color that I'm searching for
but if this were an interactive tabletop
where you could control the lighting you
could have read discs on top of a green
table or white discs on top of a black
table then this is something you could
do so I've shown you one improvement
here the first improvement is just using
the average and again as soon as I
introduce another cup you know we've got
a problem look at it that dot is right
in between them right because it's the
average location which is kind of
interesting but not what I'm looking for
so in not in the net in the next video
I'm going to look at tracking motion
which I think will be interesting to see
but the video following that I am going
to I'm going to show you how you can
track each of these locations as blobs
separately okay uh so that concludes
this particular video and in the next
video we're look at motion tracking it
frame differencing
you