I actually pulled out an older piece of work focusing just on the gesture.
This is by Nam and Wohn of recognizing hand gestures using some HMMs.
Just because it was very paradigmatic of how people would do things.
Well, what's kind of cool about their system was,
they had a bunch of different gestures that they wanted to indicate.
Like nouns.
You know,
chairs, vase, lamp, bulb because of course that's a natural vocabulary.
I have no idea.
And then you want to do things like put it down or bring it or put or
discard or jump.
I don't, don't even know why that was a particular verb that they wanted to use.
But the idea was that there was a fixed vocabulary.
The inputs that they were going to use and
they were going to control it with the hand, was a couple of different kinds.
Right? They had something to do about hand
configuration or posture, something to do with how your fingers were shaped.
Okay. They also had how you're holding your
palm and then they had a sequence of hand position, a motion, okay?
And they wanted to re, use all of this information
to make a decision about what gesture you were indicating.
So the first thing you should ask is how are they getting this information?
How are they getting this information?
>> Boy, Meghan woke up.
Well they were using something called a data glove.
Here's a picture.
So a data glove is this thing that you can wear on your hand that had
both accelerometer and gyro information about how it was being globally
manipulated and also knew something about how you were moving your fingers.
So they built this kind of interesting system.
And the reason I liked it is, they actually are using
these three sensors the angles, the orientation, the position, right?
And what's interesting is they use the HMMs
just to get information about that motion, trajectory.
Right?
They had other methods of putting into their system
how your orientation was changing and what the angles were.
Right? So the idea is that you might have some
basic overall probabilistic integration system, and in order to
make use of that, you needed things that would output the probability or
the likelihood of getting a particular movement for
a given gesture, and HMMs were exactly the kind of thing to use for that.
In fact, discriminative methods might have been harder to use for
that because they need to combine these probabilities.
So here was something very simple that they did.
They took the, I shouldn't say very soon.
Here's what they did, it required some engineering.
You take your raw data, which was a trajectory that you would move in xyz,
and the first thing they did was, they said, well you know,
your gesture is pretty much planar.
Right?
The plane could be this way, could be this way,
but the idea is that a single gesture didn't move in more than a plane.
I might even go like this, but
you'll notice even this figure 8 is in this plane.
All right?
So they find the plane that the gesture fit best to, projected down.
And then they did what's called a chain code.
We didn't actually talk about chain codes, if you did some machine vision,
you would learn about chain code.
It was a way of encoding a contour as being you know,
maybe you can only take one of eight steps.
Right? So you go up,
you go you take a direction north.
You go northeast.
You go east, east, east., south, southeast, southwest and
then eventually you come around.
And you could think of the entire contour as a sequence of discrete steps.
Right.
So north, east, south, right?
So sequence of discrete symbols, ah-hah.
HMMs.
So they trained a very simple left-to-right HMM, let-to-right
meaning that you go through every state and you, you don't skip states you have
these loopback things because you may stay in a state for a while where,
essentially you are going kind of, let's suppose you are going north, northeast.
But you didn't have that as a direction.
Well, north northeast, that state you might sometimes put out north.
Sometimes put out northeast, etc, whatever.
But you'd stay in that state for awhile.
It would output,
there would be two symbols it could output with high probability.
But that state for example would never output south.
Okay, so that would be, you'd stay in the state for a while.
So they trained up these HMMs and they did one HMM for
sort of each gesture, okay?
They also did some stuff that had to do with what they call junctures
doing transitions between gestures so that it, it could parse them.
I don't think I'll, I'll talk about that here.
I just will say that people hook up multiple HMM's together,
to form sort of an HMM that can deal with a sequence of gestures, and it's,
it's not too much more complicated than the original HMM.
So they do that and then they test it.
And because they're testing it on a very controlled vocabulary, using
features defined to work, because they want their paper to be accepted, okay?
You see results that look like this, okay?
So here are their hits.
This is the percentage of, hits are the percentage of the actual targets
that were correctly labelled, misses are when you don't label a thing correctly.
And as you can see,
they're doing tests of over a 100 of each of these examples and almost
all of their gestures are recognized with a relatively high accuracy.
You can see there are one or two that was that were not so high and
what and showing a table like this is one thing,
a slightly more interesting table might be a confusion matrix.
This is like we showed for the SVMs for the bag of words where you go back and
take a look at the slide you'll see a table.
Then it says how often is an airplane confused as a face, or
that kind of thing.
Normally if you're going to report results of that doing classification.
Don't just provide hits and misses, also provide a confusion matrix
because sometimes the actual source of the performance
is that some of the categories were much more confusable than the others.
You have no way of knowing that if you just report sort of performance
rate overall, but you need to be able to see the the, the confusion.
What, which, which element, or
which categories were confused with which categories.