All right, so we have a problem.
The brightest constraint equation gives us more unknowns than equations.
More directions to move than we have pixels.
How are we going to recover the motion?
Obviously, we need some extra constraints.
We need some additional constraints.
So, we'll talk about sort of a global approach now, and
the next time we'll talk about a more local approach that,
that's actually used and, in fact, some of you may even implement.
Some time long, long ago in a galaxy nearby Berthold Horn and
Brad Schunk did some work on what was called smooth optical flows.
And basically they said,
let's try to come up with an optical flow that minimizes some error.
Okay?
So the first part of the error is how much do we violate the optic,
the brightness constancy constraint equation, right?
Basically, that middle term right there, that's supposed to be zero.
Right? Ixu plus Iyv plus I sub t is equal
to zero.
So, violations of that means that it's not zero.
Sometimes it's positive sometimes negative.
I square it, and I sum it up over the whole image.
That would be how bad my flow is with respect to the constraint equation.
All right?
But we know we can solve that exactly because we have more unknowns
than constraints, so If that's our only constraint, we're, we're not done.
But we're going to call this the constraint error.
By the way, I kept partially originally I'd stolen, then I did on my own, I kept
the integral signs instead of summations because back when this work was done
we'd like to think about images as actually continual, continual functions.
Of course, if you were to implement this,
you'd actually have to do summations.
I'll also tell you the fact that summations are not integrals matters and
if you take a look at some of the work that was done in shape from shading,
doing it as an integral was actually hard.
Doing it in summations was, in some sense, a little bit easier.
But for here, the difference doesn't matter, so
I just wrote it as being summed up over the image.
So here we have our optic flow constraint term.
Now we need another term.
Well, it says smooth optical flow, here's the idea.
Remember when we did stereo?
The, the state of the art solution that I showed you
with that beautiful recovery?
Used a bunch of terms, some that had to do with the data, but
it also had to do with the smoothness of the disparity field?
That was the idea that surfaces don't change a lot, all over the place.
But they tend to have a particular depth or very smoothly.
The same could be said of motion, that is, so if I've got something moving
around, the motion vectors from one point to the next change slowly.
So let me come up with a penalty term that says I really don't like
velocity fields where the velocity changes quickly from one
location to the neighboring location.
So that's shown here.
These ux, ui, that's the derivative of u.
Remember, u is the x component of the motion field.
The derivative in the x direction squared, because it'd be positive,
negative, derivative in the y direction squared, same thing as here,
which is v sub x, v sub y, the change of v as you move in x,
the change in v as you move in y.
Again, everything is squared.
So the idea is, we would like this thing to be, like, zero, or
as little as possible.
By the way, what would it mean for this to be zero?
Well, for this to be zero,
that would mean that all the u's were constant, right?
They were all the same value, so the derivatives would be zero.
And all the v's would be constant.
So if all my pixels were moving with the same uv,
the whole image was moving the same direction, this value would be zero.
Any other u's and v's will cause this term to have some value,
and we want to try to minimize this thing.
So now, we have these two terms.
We have the constraint equation error term and
we have the smoothness error term called e sub c, e sub s and what we want to do
is we want to find the uv at every pixel and there's going to be a different
uv at every pixel that minimizes this total error which is just
the smoothness term plus some waiting factor times the constraint term.
And the waiting factor is sort of, how much you believe your data.
So, the bigger the waiting factor is the more important the satisfying
the brightness constancy constraint is, the smaller the waiting factor, the more
the smoothness term dominates and what value it should have is a function of
sort of how much noise there is and how smooth you believe the entire thing is.
So, this was part of a, a general approach, variational approach where
you're going to solve for uv as a function of xy that is, you're going to
find the optic flow u and v for every pixel that tries to minimize this error.
It's somewhat painful to try to implement systems that can
solve for that.
Not impossible.
But the idea is that you have a global constraint, the smoothness constraint,
to help solve your problem.