we thought about how you can talk about
optic flow with little changes in space
and little changes in time that means
derivatives and so things like sabel
which we can calculate across the image
so there's something called the optic
flow equation which basically combines
these derivatives in the image these
gradients and also in that equation are
these two things that we want to get out
this u and this v so these are our optic
flow vector for each pixel which weighs
things going in order to do that though
we have to make some more assumptions
because there's two unknowns and we have
one equation so this is where people
come up with a load of different ideas
for how to
frame the problem so that you can get at
those u and v values
so one method that um
solves for optic flow and gives you
these little what are called quiver
plots where for each pixel here you've
got this
u and v
vector so you can plot these across a
whole image and if you look in the video
that is on screen now you can see all
these plotted and showing things moving
around one of the methods for
calculating this stuff um originated in
the 80s so i think it was 1981
um
by a couple of people called horn and
shunk so if you look up optic flow this
will be the sort of one of the first
techniques that's mentioned um
again so bear in mind that in in the 80s
it was a challenge just to get any kind
of video onto a computer at all um
and uh yeah you know you're talking
about working with tv standards like pal
or ntsc rather than things like vga
which came along uh sort of nearer near
the 90s so these are going to be
originally quite low resolution images
maybe
320 by 200 or something like that if
you're talking about 4k images here
nowadays yeah you've got a lot more
calculations to do so the assumption
that
horn chunk make
in order to get out these unv components
of optic flow is that they look in the
the local neighborhood so as well as
just considering an individual pixel so
we haven't got enough information there
to figure out what optic flow is
what they do is they say actually let's
look at the neighboring pixels here and
we're going to
basically put in a constraint into our
solver that says r u and v here should
be quite similar to an average u and an
average v in the local neighborhood what
we're going to assume is that this
motion isn't going to radically change
between pixels
if we're on a surface the motion change
is going to be small unless you're at an
edge of course so we break another
constraint there but most of the time
you know if this table's moving around
all the pixels are going to be moving in
roughly the same direction
so we have this smoothness constraint
that we build in
and horn chunk is a global approach so
for every single pixel in the image
it will
try and optimize for working out a umv
in the local pixel and comparing that to
the the average umv in the in the local
area so it's this iterative scheme so
it's pretty slow where it goes through
and tries to kind of globally
find the best unv
to fit the image it looks to me like
you've got to work out if something
that's the same is has now moved into a
different area is that yeah is that what
you're kind of like looking at it's it
is that um but remember that the motion
that we're talking about is really tiny
so we're not talking about something
that's moved forward 10 pixels that will
break optic flow although we could talk
about how you can fix that at the end
because there's one quite neat trick
you can do if something's moved really
fast
we're talking about tiny movements so
almost kind of sub pixel pixel level
movements here
so it's picking up changes in brightness
patterns spatially but also
over time as well and all the equations
do is figure out a way to get these
estimates of
of where that little change in
brightness has gone across the whole
image in the case of horn chunk horn
chunks a global method is trying to do
everything across
the whole image so there's a load of
other approaches that will calculate
this
um another really common one is called
the lucas canaday um approach to solving
it
and rather than trying to
say look let's let's optimize this thing
globally across the whole image
they look at a little patch
so they take a patch of pixels so again
this looks a bit like a kernel i guess
so it's normally sort of five by five
and you see you're considering the pixel
in the middle
but what the lucas canada approach says
is um let's assume that u and v is going
to be the same in all of these pixels in
this region and that gives us 25
equations which is over determined so
you can use least squares to figure out
the best fit essentially for you and v
there
and of course all these things are
making a huge load of assumptions that
which i've already hinted uh we have to
break quite a lot so if you've got an
edge here for example that maybe this
object is moving that way and this
object is moving that way
you're going to have problems figuring
out a umv there so some constraints
might build in things that try and
separate out edges
because it it tends to break this stuff
another quite
interesting
problem that you get with some of these
methods is something called the aperture
problem so this is where we're trying to
figure out motion so it's called the
aperture problem because we've only got
a little window that we can see motion
happening in like that so the question
is which way is that line moving so if
we had to put optic flow
vectors on this line where would you say
it was going well i mean the obvious
thing to say is it's going down but it
could of course be a diagonal line
moving right it's a good answer because
it can be lots of things if we take away
that window this is the motion that
we're actually getting so the line's
just moving across the image from left
to right but it looks there that it's
kind of either going diagonally down
right or down kind of depends on how you
interpret it i guess this is called the
aperture problem or
the barbershop pole illusion
because it's got stripes moving up and
down and the idea being that there's not
enough information here to um
to accurately figure out how that that
feature is moving it's very easy here
because we can see the corners so
corners are a bit special
and they allow us to
sort of
refine our estimates of motion um so
sometimes if we're only looking in a um
in a small window
like here
we can get ambiguous motion happening
that we can't uh determine because of
things like the aperture problem so it's
another sort of issue with these these
methods the only other thing that i
wanted to mention here is that um so
horn chunk is global we talked about it
kind of finding the best set of umvs
across the image
this approach here is local so we only
care about making it work on a five by
five
patch of pixels essentially but uh
they've got advantages and disadvantages
so one of the advantages of the global
approach is if we've got an object here
um that's moving at the edges you know
there's enough brightness changes that
we can pick up movement happening but
there's a question about if this is just
a sort of orange or white or whatever
shape what's happening in the middle of
it it's like the spinning ball we can't
tell so the nice thing about a global
approach is it will kind of fill in
from the information it knows it will
fill in the edges throughout the shape
okay
and the problem with the local
approaches is if you've just got a patch
here
yeah you can kind of figure out a umv
in this location but if your patch is in
the middle of one of these textureless
shapes it's kind of an undetermined
solution so you might get some sort of
noisy approaches
um
so it swings and roundabouts um as with
all the all of this stuff as to whether
you use a global approach or a local
approach and they've got trade-offs and
speed and things like that
that's just two examples there's loads
of different ways of calculating optic
flow people are doing it with deep
learning now as well of course uh
um so lots of different ways of doing it
and it's still even though it's been
around since the 80s it's
it's a very um useful technique still as
a way of pre-processing things perhaps
as a precursor for segmentation so if
this shape here and the background are a
very similar color or texture but this
shape's moving and the background's not
moving so you've not got any flow
vectors on the background you can use
the optical flow field as a way of
segmenting what's going on
okay so uh we've said that the motion
has to be really small for any of this
to work so you need a really small time
between frames
um again another assumption that's going
to get broken all the time is that stuff
moves more than a few pixels so if
you've got an image that looks like this
and you've got something here and in the
next frame it moves down here
optic flow's not going to like that and
it's going to break so you're not going
to get a good value out for that there's
a trick called
building an image pyramid
which certainly the lucas canada
approach uses so you might read about
this approach using a pyramid scheme and
all that means it's actually pretty
simple
you make your image lower resolution to
start with
so
perhaps if i switch to a different color
imagine if instead of being a four by
four
pixel image
this is a two by two pixel image and
then whatever shape we've got here yeah
okay so it averages out a bit because
we're sort of blurring it with our
surrounding ones but now they've become
neighboring pixels and we've essentially
shrunk the space over the motion that's
happening so you end up with this
pyramid
sort of system where you have
low resolution
lower down
and then you kind of move up to higher
and higher resolutions that's a terrible
image do you then have to average out
and say okay that one pixel is yeah the
same for 20 pixels or exactly so you use
this as a way to kind of bootstrap the
rest of it so you calculate your motion
here
so you get your motion vectors that
might look like this
on the low resolution one and
essentially you populate the next level
up with estimates of the motion from
these so you know whatever was here
gets filled into these four pixels and
then you do the scheme again but because
you've got a starting point this time it
will help you sort of overcome some of
those bigger motions is this being used
these days you mentioned deep learning
what what sort of things is it used for
at the moment uh yeah it's used today so
um
i mentioned it can be used for image
stabilization so you can stabilize an
image by looking at how the pixels are
moving around
you don't have to calculate it across
the whole image if you want to do it
really quickly you could just kind of
sample bits of it but you want to get an
idea of is the camera moving around the
world in some way and then you can sort
of in software correct for that
another use is
frame interpolation so if you've got
um 25 frames per second video and you
want to turn it into kind of fake
slow-mo
if you just stretch out your frames it
will go all kind of jaggedy right so you
get a frame and then the next frame and
because you've filmed it at normal speed
and you're slowing it down it doesn't
look very nice so if you know how things
are moving um across the frame
you can add in sort of fake extra frames
so if this is your first frame and your
second frame you can add in additional
frames in the middle
which you can use optic flow to kind of
figure out
how brightness is moving around at that
point and if you know how the sort of
local patterns are moving about you can
put them in a sensible place in those
interpolated frames so rather than just
pure sort of smoothing or interpolation
between them it's kind of a bit more
sensible than that a bit cleverer and
you can
move surfaces sort of where they should
be so there's some quite sort of neat
plug-ins and tricks coming around doing
that kind of stuff and one way you can
do that is using optical flow if you're
trying to follow something moving very
fast in an image you know a thing rather
than just talking about movement at the
pixel level that's going to be where
you're you're looking at object tracking
which perhaps we could do a video on in
the future
so for a number of reasons if you've got
a wobbly shaky camera you can use it for
image stabilization
i wasn't insinuating anything um yeah so
you can use optic flow to see what kind
of global motion poker blur i don't know
how to pronounce that word bokeh is
right yeah bokeh right bokeh bouquet