Gesture recognition is a topic in computer science and language technology with the goal
of interpreting human gestures via mathematical algorithms. Gestures can originate from any
bodily motion or state but commonly originate from the face or hand. Current focuses in
the field include emotion recognition from the face and hand gesture recognition. Many
approaches have been made using cameras and computer vision algorithms to interpret sign
language. However, the identification and recognition of posture, gait, proxemics, and
human behaviors is also the subject of gesture recognition techniques. Gesture recognition
can be seen as a way for computers to begin to understand human body language, thus building
a richer bridge between machines and humans than primitive text user interfaces or even
GUIs, which still limit the majority of input to keyboard and mouse.
Gesture recognition enables humans to communicate with the machine and interact naturally without
any mechanical devices. Using the concept of gesture recognition, it is possible to
point a finger at the computer screen so that the cursor will move accordingly. This could
potentially make conventional input devices such as mouse, keyboards and even touch-screens
redundant. Gesture recognition can be conducted with
techniques from computer vision and image processing.
The literature includes ongoing work in the computer vision field on capturing gestures
or more general human pose and movements by cameras connected to a computer.
Gesture recognition and pen computing: This computing not only going to reduce the hardware
impact of the system but also it increases the range of usage of physical world object
instead of digital object like keyboards, mouses. Using this we can implement and can
create a new thesis of creating of new hardware no requirement of monitors too. This idea
may lead us to the creation of holographic display. The term gesture recognition has
been used to refer more narrowly to non-text-input handwriting symbols, such as inking on a graphics
tablet, multi-touch gestures, and mouse gesture recognition. This is computer interaction
through the drawing of symbols with a pointing device cursor.
Gesture types In computer interfaces, two types of gestures
are distinguished: We consider online gestures, which can also be regarded as direct manipulations
like scaling and rotating. In contrast, offline gestures are usually processed after the interaction
is finished; e. g. a circle is drawn to activate a context menu.
Offline gestures: Those gestures that are processed after the user interaction with
the object. An example is the gesture to activate a menu.
Online gestures: Direct manipulation gestures. They are used to scale or rotate a tangible
object. Input devices
The ability to track a person's movements and determine what gestures they may be performing
can be achieved through various tools. Although there is a large amount of research done in
image/video based gesture recognition, there is some variation within the tools and environments
used between implementations. Wired gloves. These can provide input to the
computer about the position and rotation of the hands using magnetic or inertial tracking
devices. Furthermore, some gloves can detect finger bending with a high degree of accuracy,
or even provide haptic feedback to the user, which is a simulation of the sense of touch.
The first commercially available hand-tracking glove-type device was the DataGlove, a glove-type
device which could detect hand position, movement and finger bending. This uses fiber optic
cables running down the back of the hand. Light pulses are created and when the fingers
are bent, light leaks through small cracks and the loss is registered, giving an approximation
of the hand pose. Depth-aware cameras. Using specialized cameras
such as structured light or time-of-flight cameras, one can generate a depth map of what
is being seen through the camera at a short range, and use this data to approximate a
3d representation of what is being seen. These can be effective for detection of hand gestures
due to their short range capabilities. Stereo cameras. Using two cameras whose relations
to one another are known, a 3d representation can be approximated by the output of the cameras.
To get the cameras' relations, one can use a positioning reference such as a lexian-stripe
or infrared emitters. In combination with direct motion measurement gestures can directly
be detected. Controller-based gestures. These controllers
act as an extension of the body so that when gestures are performed, some of their motion
can be conveniently captured by software. Mouse gestures are one such example, where
the motion of the mouse is correlated to a symbol being drawn by a person's hand, as
is the Wii Remote or the Myo, which can study changes in acceleration over time to represent
gestures. Devices such as the LG Electronics Magic Wand, the Loop and the Scoop use Hillcrest
Labs' Freespace technology, which uses MEMS accelerometers, gyroscopes and other sensors
to translate gestures into cursor movement. The software also compensates for human tremor
and inadvertent movement. AudioCubes are another example. The sensors of these smart light
emitting cubes can be used to sense hands and fingers as well as other objects nearby,
and can be used to process data. Most applications are in music and sound synthesis, but can
be applied to other fields. Single camera. A standard 2D camera can be
used for gesture recognition where the resources/environment would not be convenient for other forms of
image-based recognition. Earlier it was thought that single camera may not be as effective
as stereo or depth aware cameras, but some companies are challenging this theory. Software-based
gesture recognition technology using a standard 2D camera that can detect robust hand gestures,
hand signs, as well as track hands or fingertip at high accuracy has already been embedded
in Lenovo’s Yoga ultrabooks , Pantech’s Vega LTE smartphones, Hisense’s Smart TV
models, among other devices. Algorithms
Depending on the type of the input data, the approach for interpreting a gesture could
be done in different ways. However, most of the techniques rely on key pointers represented
in a 3D coordinate system. Based on the relative motion of these, the gesture can be detected
with a high accuracy, depending of the quality of the input and the algorithm’s approach.
In order to interpret movements of the body, one has to classify them according to common
properties and the message the movements may express. For example, in sign language each
gesture represents a word or phrase. The taxonomy that seems very appropriate for Human-Computer
Interaction has been proposed by Quek in "Toward a Vision-Based Hand Gesture Interface". He
presents several interactive gesture systems in order to capture the whole space of the
gestures: 1. Manipulative; 2. Semaphoric; 3. Conversational.
Some literature differentiates 2 different approaches in gesture recognition: a 3D model
based and an appearance-based. The foremost method makes use of 3D information of key
elements of the body parts in order to obtain several important parameters, like palm position
or joint angles. On the other hand, Appearance-based systems use images or videos for direct interpretation.
3D model-based algorithms The 3D model approach can use volumetric or
skeletal models, or even a combination of the two. Volumetric approaches have been heavily
used in computer animation industry and for computer vision purposes. The models are generally
created of complicated 3D surfaces, like NURBS or polygon meshes.
The drawback of this method is that is very computational intensive, and systems for live
analysis are still to be developed. For the moment, a more interesting approach would
be to map simple primitive objects to the person’s most important body parts and analyse
the way these interact with each other. Furthermore, some abstract structures like super-quadrics
and generalised cylinders may be even more suitable for approximating the body parts.
The exciting thing about this approach is that the parameters for these objects are
quite simple. In order to better model the relation between these, we make use of constraints
and hierarchies between our objects.
Skeletal-based algorithms Instead of using intensive processing of the
3D models and dealing with a lot of parameters, one can just use a simplified version of joint
angle parameters along with segment lengths. This is known as a skeletal representation
of the body, where a virtual skeleton of the person is computed and parts of the body are
mapped to certain segments. The analysis here is done using the position and orientation
of these segments and the relation between each one of them( for example the angle between
the joints and the relative position or orientation) Advantages of using skeletal models:
Algorithms are faster because only key parameters are analyzed.
Pattern matching against a template database is possible
Using key points allows the detection program to focus on the significant parts of the body
Appearance-based models These models don’t use a spatial representation
of the body anymore, because they derive the parameters directly from the images or videos
using a template database. Some are based on the deformable 2D templates of the human
parts of the body, particularly hands. Deformable templates are sets of points on the outline
of an object, used as interpolation nodes for the object’s outline approximation.
One of the simplest interpolation function is linear, which performs an average shape
from point sets, point variability parameters and external deformators. These template-based
models are mostly used for hand-tracking, but could also be of use for simple gesture
classification. A second approach in gesture detecting using
appearance-based models uses image sequences as gesture templates. Parameters for this
method are either the images themselves, or certain features derived from these. Most
of the time, only one or two views are used. Challenges
There are many challenges associated with the accuracy and usefulness of gesture recognition
software. For image-based gesture recognition there are limitations on the equipment used
and image noise. Images or video may not be under consistent lighting, or in the same
location. Items in the background or distinct features of the users may make recognition
more difficult. The variety of implementations for image-based
gesture recognition may also cause issue for viability of the technology to general usage.
For example, an algorithm calibrated for one camera may not work for a different camera.
The amount of background noise also causes tracking and recognition difficulties, especially
when occlusions occur. Furthermore, the distance from the camera, and the camera's resolution
and quality, also cause variations in recognition accuracy.
In order to capture human gestures by visual sensors, robust computer vision methods are
also required, for example for hand tracking and hand posture recognition or for capturing
movements of the head, facial expressions or gaze direction.
"Gorilla arm" "Gorilla arm" was a side-effect of vertically
oriented touch-screen or light-pen use. In periods of prolonged use, users' arms began
to feel fatigue and/or discomfort. This effect contributed to the decline of touch-screen
input despite initial popularity in the 1980s. In order to measure arm fatigue and the gorilla
arm side effect, researchers developed a technique called Consumed Endurance.
References
External links Annotated bibliography of references to gesture
and pen computing Notes on the History of Pen-based Computing
The future, it is all a Gesture—Gesture interfaces and video gaming
Ford's Gesturally Interactive Advert—Gestures used to interact with digital signage
Kinetic Space—3D Gesture Recognition, Training & Analysis
Look Ma No Trackpad—Gestures used to control Spotify & iTunes
The Gesture Recognition Toolkit—An open-source toolkit for gesture recognition