[MUSIC PLAYING]
When talking with each other, we often
communicate with a complex mixture
of speaking and gesturing.
People are able without apparent effort
to produce rich and carefully coordinated
multimedia presentations to describe their ideas.
Gestures that occur with speech have many different functions,
from directing the listener's attention
with pointing gestures to the simple movements that
help define the rhythm of the conversation.
We are currently working on adding gestures
to the human machine interface.
The specific focus of this video is
our work on enabling the use of depictive gestures,
those gestures used to describe the spatial and/or temporal
aspects of an action or scene.
Depictive gestures do not have a well-defined standard form.
They cannot be matched to a particular model
and interpreted automatically.
Instead, this class of gesture is created on the fly,
depicting any spatial and temporal aspect the speaker
decides to describe.
We have developed a prototype system that allows a user
to describe a scene through a free mixture of speech
and depicted gestures.
Create a room.
Place a table here and a chair here.
Add a mouse next to the table.
Turn the mouse and move it this way.
Move the chair.
On the table, place a teapot.
Add a glass next to the teapot.
Turn the teapot.
Look at the room from here.
Tilt the floor like this.
The user is able to communicate complex spatial relations
and actions through a simultaneous mixture of speech
and gestures.
This is not a command language interface
in which hands serve as a redundant input
mode for commands.
This is not direct manipulation where
the hands serve as a kind of three dimensional mouse.
This is a descriptive interface that
takes advantage of people's abilities
to communicate ideas through a free mixture of speech
and of gestures.
The prototype system is composed of four principal parts.
A gesture module gathers the position and posture
data for the user's hands from a pair of data gloves.
It then analyzes and segments the stream
into gestural features.
This intermediate encoding of gestures
focuses on the characterization of postures, positions
and orientation of the hands during significant static and
dynamic periods.
A speech module receives a stream
of recognized words from a connected speech recognizer.
The sequence of words is parsed to produce an expression
of semantic frames.
These frames encode the content and structure
of the user's speech.
An object base manages the objects and their appearance
on the display.
Object descriptions within the object base
include part shapes, major axes and spatial relations
among objects and parts in the displayed scene.
An interpretation module receives information
from the other three modules.
Again, gestures are not interpreted
automatically based on a match to some standard template
but rather, are assigned a meaning in the wider
context of information from speech and the object base.
A spatial representation is used to encode
both the depict of gestures and the objects in the object base.
This spatial representation allows
us to map the relations and actions depicted in gestures
to the objects in the displayed scene.
For example, when the user said move the chair like this,
the gesture module analyzes the posture and movement
of the hand.
From speech, we determined that a movement operation is
being requested for the chair.
The intermediate encoding of the user's gesture
can now be analyzed and mapped to the chair
to fill in the details of that movement.
The action is then carried out by sending
the corresponding manipulation command to the object database.
Later, the user defines the context for subsequent actions
by saying on the table.
A mapping is established between the corners
of the table depicted in gesture space
and the corresponding corners in graphical space.
Placing objects on the table can now
be performed and interpreted in the spatial context.
[MUSIC PLAYING]
This prototype demonstrates an alternative approach
to gestures at the human computer
interface, one in which the user does not learn an arbitrary
set of signs.
But rather, allows the user to freely mix speech and gesture
to describe the spatial and temporal aspects of a scene.
Creating descriptions where speech and depictive gestures
complement each other represents a skill we already possess
and a potentially powerful language
for interacting with graphical domains.
[MUSIC PLAYING]