So, I'm pretty sure we showed this before when we were talking about motion
and perceptual organization.
And as it says here,
even an impoverished motion data can evoke a percept, and
presumably, you all see these moving dots as a person walking, right?
So, you're not recognizing this as a person from the appearance of the dots.
You're recognizing this as motion and then you see that it's a person.
And in fact, I can prove that to you looking at some work that was done by
myself and Jim Davis now professor at in Ohio.
And so I'm going to show you here.
So when you take a look at this picture,
unless you've seen this demo before, you probably don't know what that is.
When I put this thing in motion, what do you see?
What you all probably saw, was somebody sitting down.
And what's interesting is, now when the video is done, you can sort of see that,
oh, this is a person standing up right here, right?
In fact, there's some sort of a blackboard behind them, and
then once they sit down, they're head is here and the body is here,
because there's actually a chair over here.
But until that motion happens, you can't actually recognize that.
So we did a little bit of work on some technology that would let you
recognize this, and we created this thing called motion energy images.
So here you see a picture of a person, this is a clear video,
all right, where the person is sitting down.
That's, by the way, Claudio Pinhanez, who also got a PhD in Computer Vision,
now doing great stuff down in Brazil.
And what these are representing are cumulative images of how,
of where stuff has moved, okay?
So the idea is, when you're all done, this is area in which stuff has moved.
And, you know, the question is, that's somehow indicative,
that something about that pattern and
how it moved in there is indicative of the actual motion pattern itself.
So, can we recognize that explicitly?
So we created this thing called motion history images which
becomes the motion energy images also.
Which are a function of the temporal volume, and
what I mean by temporal volume is that same thing, right, x, y, and t.
So I've got a collection of video.
And it's really a very simple recursive computation.
This is an MHI image, motion history image, and
the way it works is quite simple.
As each new frame comes in, in this particular case you run some algorithm
to tell me, which pixels are moving in that frame.
Now, of course, this is under the assumption of a static camera.
And if you're not using a static camera, you'd have to compensate for the motion
of the camera, threshold out the motion of the person, all the usual caveats.
But then when you do that, you get this very simple operation.
If a pixel is moving, set that location at time t,
to some maximum value tau, and you're going to pick tau.
Maybe tau is 15.
I'll tell you why 15 in a minute, right?
And if it's not moving, that's the otherwise, I'm just going to decrement,
whoops, over here, decrement the value, okay,
from the previous time, but I don't go any less than 0.
So the image on the right is an example.
The brighter pixels like over here, they've moved more recently.
So this is time t, maybe they've moved at time t minus 1,
they just moved before.
So they started out as tau.
So maybe tau is 15x, let's make it 30, that example will work better.
It was 30, so now it would be 29.
But the pixels up here, they move, the last time they moved is 15 frames ago.
So its value would be now, tau minus 15, so if tau were 15,
it would be 0, so that would be black, that's why it didn't work so well.
But if tau were 30t minus 15, it would be now down to 15, right?
It would have counted down to 15.
So these areas are nice and bright white because they just moved.
These areas are darker gray because, it was a longer time ago that they moved.
And so the way you can think about this motion history image,
is that it both shows you sort of where stuff is moved.
And it also shows you sort of how stuff is moved,
because you can tell that the stuff up here moved a long time ago, and
stuff down here is moving more recently.
It's a recursive filter, that is,
I only have to keep track of i of tau of x, y, and t minus, of t.
And I can create the new one when the next frame comes in.
So I don't have to remember my old frames.
The other thing is, where did tau come from?
Well, the pixels go from tau down to 0.
So, let's suppose I'm bringing in 30 frames a second.
So if I set tau to 30, then from 30 to 0 is the range over a second.
And what that would mean it that
anything that happened longer than a second ago, I'm not worrying about.
So I'm recognizing action that's length of a second.
Suppose you wanted to check whether something could happen slowly, right?
Like, he sits down slowly,
it takes him a full second to sit down because he's really slow.
Or somebody sits down quickly, right, because they go like this and
they, it's not Megan playing with the video.
Anyway so they can do it in a tenth of second, so sorry, in a, in, in a,
a third of a second, so that'd be tau of ten, right, 10 frames.
Well one of the kind of cool things about motion history images
is if you give me the motion history of image of tau, right?
So everything goes from let's say 30 to 0.
I can just, if I wanted the motion history image for tau equal to 20,
I could just subtract 10 from it, right, so 30 becomes 20.
And anything that's less than 10 just becomes 0.
So it only looks back 20 frames.
So it's trivial to compute I I of tau 20
from motion history tau 20 from motion history tau 30.
So I only have to keep one around, whichever one's the longest one, and
I can check all of those time windows at the same time.
And then also as should be clear,
if this is the motion history image that goes from white to dark.
If I just wanted the motion energy image where anything has changed,
all I can do is just threshold that picture, right?
Anything greater than 0, make a 1, otherwise it's a 0.