today i thought we'd talk about
something called optic flow
so this is something that you might not
have heard about before but it actually
turns up in quite a lot of different
things so very broadly optic flow is
about understanding
how things are moving in an image so
at the pixel level
how motion is happening so how things
are moving around across the whole image
and this is useful for a number of
reasons if you've got a wobbly shaky
camera you can use it for image
stabilization i wasn't insinuating
anything um
yeah so you can use optic flow to see
what kind of global motion is happening
and sort of take that out
drones things like that some of them
have got optical sensors that will kind
of look at sort of how the world's
moving around them so this isn't about
tracking sort of individual objects or
things this is about seeing how the
global motion all the surfaces in the
scene are moving around
if you can kind of break up the scene by
motion
it's another way of segmenting an image
or labeling an image
if you've seen some videos about that
before so
in addition to things like color or
texture you can use how things are
moving around to help sort of figure out
how to divide up a scene so like almost
like 3d i suppose if you get a parallax
if i'm yeah exactly across here you can
see that you're moving differently to
the window
exactly and you can other you can pull
out some other things there as well so
things closer to the camera tend to move
faster so you get kind of an idea of
depth and there's certain techniques
such as something called structure from
motion which does exactly that it
figures out how things appear in 3d by
seeing how the camera moves around let's
have a think about sort of what's going
on in an image as you move a camera
so uh obviously an image is composed of
loads of pixels potentially millions of
pixels and when you move your camera
around there's some kind of change
happening in those pixels
and what we want to figure out is uh how
that motion at an individual pixel level
is happening okay so that means
pretty much for every pixel in the image
we want to try and apply
some kind of motion vector to it to
understand where it's moving to okay now
this is a pretty hard problem
there's many many pixels
in the image and to understand exactly
how each of them are moving we're going
to have to kind of make some assumptions
and some simplifications and that's
where
the optic flow techniques kind of come
in so this is a really simple image here
where we've just got sort of one shape
in the middle of it like this perhaps a
ball or something like that and if we're
calculating optic flow across this image
and the ball is moving what we're going
to end up with is remember this is sort
of every single pixel we're going to get
these flow vectors when we're talking
about optic flow
really what we're considering is sort of
the motion at this level
within the object or surface that's
moving what's going on okay so here i've
drawn some pretty bad optic flow vectors
showing that maybe this ball is sort of
moving down and if the background was
moving as well you know perhaps you'd
have lots of vectors over the background
showing that the background was moving a
certain way so different
surfaces in the image can move in
different directions
but it's important to remember that
we're just working at the pixel level so
we're trying to
pick apart what's happening on
individual pixels within the image so
this is um quite complicated to do
because the real world is complicated so
we have to make a lot of assumptions in
order to try and pull out the motion of
essentially these individual pixels that
represent something called motion flow
and that's something sort of moving
through the real world so if i move
something around that's moving in the
real world but how does it move on the
image we've just got this 2d plane of
pixels and so how do we understand from
that how something's moving sort of in
3d
so there's a number of assumptions that
we have to make
one of the assumptions that we have to
make um is that the things like the
lighting doesn't change okay so if you
if you think of this piece of paper if i
move my hand across it and we get a
shadow
then optic flow techniques might pick up
on that shadow as being something
actually moving in the image rather than
just the lighting changing
so one thing we have to assume is that
lighting isn't isn't changing too much
and obviously in the real world that's
quite challenging because we get lots of
shadows and
and stuff happening that we it's going
to confuse these kind of approaches
um so another
sort of simplification that we make is
that when we're looking at this flow we
only do it over very small changes in in
time so unlike tracking where we might
be following something through a whole
video sequence here we're just looking
at two frames so we'll take two
neighboring frames out of a video and
see how the pixels have moved between
them as well as having things like
lighting changing which can break optic
flow
another problem we can get sort of the
other way around is that maybe there's
not enough features on a surface to be
able to detect the difference anyway i
was going to show you an example with a
ball but the only really shiny ball i
could find was this tiny one so if you
spin it
it's spinning of course but you can't
see it spinning because you can't see
the texture turning around on the ball
especially because it's reflecting
lighting which isn't moving so one of
the problems we have with optic flow is
that we're making all these assumptions
you know such that we can actually see
things moving and that's not always true
one of the key constraints we make when
we're writing down how we're going to
calculate optic flow is this idea of a
brightness constraint and what that
really means is if we've got our image
here like this
we've got the intensity at a particular
point so this point here our brightness
in our image so our grayscale value so
the higher it is the nearer is to white
the lower is than here it is to black is
that a position x comma y so that's our
pixel at a particular time t
and what we're thinking about is
essentially where has the thing at that
pixel here move to in the next frame so
it's going to have moved only a little
bit
even much smaller than that actually but
let's draw it like that so we've got our
new position here that we think this
information has moved to whatever's in
the real world has moved here uh in the
image and so this is going to be
essentially
the brightness at x
plus a little bit y plus a little bit at
t plus a little bit
so we've got some changes happening in
space x plus a little bit y plus a
little bit and also a change happening
in time as well so because we've got
these little changes you can use
actually derivatives you can use some
neat tricks where we're just looking at
gradients
in the image to figure out
this equation we can rewrite it as
something called the optic flow equation
which just basically uses image
derivatives to look at changes in
grayscale
over very tiny patches of course if you
want to calculate derivatives in an
image you can use
things like kernels like the sabel
kernel which i think mike's done a video
on before so
the maths boils down to essentially
being able to do pixel-wise calculations
to deal with grayscale so looking in
neighboring
regions to see changes in space and
looking at changes over time as well so
differences between
uh frames i'm looking at that now from
my videographer head and thinking
4k 60 frames per second yeah this is a
lot of calculations right so there's a
few tricks
so perhaps that leads nicely on to how a
few people have solved this because this
is the problem and actually now you've
got a load of different optimization
techniques that will try and give you
these vectors where we've moved to here
this is kind of known as umv so these
are our motion vectors that we've got
everywhere in the image and you're right
it can take a little while to calculate
that
so we've thought about how
you can talk about optic flow with
little changes in space and little
changes in time that means derivatives
and so things like sabel
uh which would be if you had a camera
with a fairly high iso settings you had
a lot of grainy noise that's going to
produce a sort of low level edge over
everything