These neighboring troughs and Peaks as a
Good approximation of where the finger starts and ends
And then we can trace a long ways to finally create a hand pose and that hand pose we can then use as a driver
For input for games so we can work out if the things are extended if they're closed
The system just uses the Nintendo Wiimote from the way it connects to the computer via bluetooth for
Markers on the end of the fingertip so then picked up by a camera on the Nintendo Wiimote
It's an infrared camera and the system
Utilizes a library developed by Brian Peake called Wii mote live which is freely available via CodePlex
We built the three-dimensional tracking capability on top of that library
it uses plane of holography to be able to work out where the points are in 3d space so you have a plane at the
front and a plane at the back and we do an
Intersection of lines to each of the LEDs via those two planes and we can work out where things are in 3d there are four
LEDs on the end of the fingers they are powered by this unit on the back here
essentially it's a USB rechargeable
Unit as well when the kits out in the home that you can just plug it into the PC and charge it via the PC
The Glovers has been designed so we can swap for different sized finger pieces so we can detach and reattach it up there
So for different users we can use different lengths of wire we can also use
Individual finger attachments as well the games, which you can see on on the screen are essentially they're produced in
Microsoft's XNA framework
Which is a game development framework, which people can also use to distribute to Xbox 360?
Around 15 million people have a stroke every year the kind of movements people have to make post-stroke off pronation supination
reach to grasp and then grasp
Pick an object up and put it down in another place have the peg test where you have to pinch pick up and drop down
and what we've done is we've mapped that interaction to
Pinch and release like so so if I just show you that now
Essentially what I do is I pinch
to activate the ball, and then I release to
Defy the ball. There's a fun and interesting project to do the main difficulties
I would say would be actually building a system, which can be deployed into the home so rather than having a system
Which just works in a research lab we had to actually make it so it was robust enough to work in the real world
Environment and that's where the that
The difficulty came in as far as coding and that sort of thing goes so Paul the problems that we had with this system were
Things like a phantom IR reflection so this works off a series of infrared emitting diodes. We're getting other
Infrared sources from around the house such as perhaps a television remote if you're wearing a glove there might be
Patient discomfort there might be hygiene issues you might find it difficult to get the glove on there's lots of little
technical parts on the glove which can get broken, so we're looking at whether we can use the Kinect camera as a
Marker, 'la system where you don't have to wear anything at all for fast and robust tracking in a home environment or what we've got
in front of us now is essentially a hand tracking system that takes advantage of the depth map features of the Kinect camera so
in computer vision research
Traditionally has been a lot of background in using color based cameras to extract people's hands and hand pose
information from a scene
And they work really really well in a tightly controlled lab setting so when you've got things like green screen
very tight control of the lighting conditions
But that doesn't translate well into a real home setting and a practical kind of view scenario for a home-based drug rehabilitation
system
Now the advantage of the Kinect is that instead of giving you a color feed
Yeah, actually gives you a depth map of the scene so instead of having color information encoded in pic
encourage the distance in millimeters of each pixel in the scene
And it's much more tolerant to environmental conditions and things like skin tone
This is a proof-of-concept game designed to test the tracking system, so it's based on the slingshot game
we saw earlier using the Wemo and this is designed to try the
exercise and arm synergy
So we're using the shoulder elbow that reach and the pinching grasp the game randomly positions a ball on the screen
And you just need to move my hand over the ball
grab it and throw to the target and then a ball will reappear, and we've got kind of a
Developer mode to help us see what's going on so you can see it's finding kind of that large red circle
Which is really the key element of the hand tracking works trying to find the center of the palm
We then try to look at nearby information to determine fingertips pits of the fingers
and we can use that to
Eventually create these kind of purple lines that we can trace along to find the depth information about the fingers and from that
calculator hunt pose
We essentially locate the hand in the scene segment all the other informations remove everything
We're not interested in background your arm your face so you've only got the hand left on the depth map
We then trace around that hand contour
So you create kind of a contour route around the outside of your hand and this helps account for a lot of the missing information
So you might have blank spots within the end, but because we're now only looking at the contour
We kind of eliminate all those. We then try and find the largest area
We're inside that contour because that invariably turns out to be the palm of the hand we then do some distance
analysis of the contour away from the center of the palm we do some filtering to smooth that out and this helps us classify a
distance graph into peaks and troughs and we then classified maximal points as fingertips and
Minimal points is finger troughs and we can simply use neighboring troughs and Peaks as a good
Approximation of where the finger starts and ends and then we can trace along ways to finally create a hand pose and that hand pose
We can then use the driver input for games so we can work out if the things are extended if they're closed
Ok so connect is a novel database camera, so it uses a structured lights approach
So instead of actually just being one camera that you might think of traditionally it has an infrared projector
So it's projecting a pattern in infrared light onto
Everything in the scene that we can't see it then uses an infrared camera to look at that pattern and it looks for
Distortions in the pattern and eventually uses triangulation to calculate the depth in the scene the downside to that is that it's not
Particularly accurate we have done some experiments to try and verify the accuracy because we're using it for clinical purposes
also things generate a lot of missing information especially on small
objects which is quite a big problem when you're looking at something like the hand because
Fingertips can quite often disappear now a time-of-flight camera like this one
I have here has traditionally been a lot more expensive so this cameras in the region of several thousand pounds to buy
But this actually projects infrared light and measures the time it takes for it to travel back into the camera
So that creates a much more accurate
Depth map because it's not reliant on other information and the size of objects within the scene the connect to camera is
Reported to use a very very similar coming to this I believe the camera develops even by the same company
So we're really excited to see kind of what?
We'll be able to do move the same kind of technology and low-cost
Camera as a Kinect. That's definitely suitable for home use because it's designed for playing games in the home
but with the kind of the
Accuracy we've got ustah from these kind of cameras a camera like this which traditionally cost between two to four thousand
Pounds so they these are the kind of cameras that have been applied for industrial use in production lines and monitoring
But the Kinect really revolutionized the depth camera scene so it was the the first kind of real low-cost consumer
Available that camera available, and it's had a huge impact on computer vision
Technology just getting better and better
And I think it will make a huge impact not just on our research team bond the kind of that this whole field in general
Connect to will be out at Christmas with the launch of the new
Xbox one
but we're hoping that we can get on the
Developer list so there is a list of people to get the SDK early if they're doing interesting enough projects so hopefully you
I think it realizing something like photon shift that really I'm more interested in using it then I
Leave that to physicists to work out and it's where they can write some code show me your code
And I can give you a job