Hello, I am Tianyi Wang from Purdue University. In this video, I introduce our work, GesturAR,
an authoring system for creating freehand interactive AR applications.
I will first talk about the background and design space.
Then I will explain the workflow and the gesture detection methods.
Finally, I will go through some application scenarios that were created by our system.
Empowered by hand tracking techniques, freehand interaction has become an important way to
interact with virtual content. But most AR systems only support limited gestures
such as pointing and grabbing. Some prior works invited end-users to customize
the hand interactions. Yet these works involved an offline learning
process and users could not test and play with the interactions immediately.
Some other works utilized the programming by demonstration metaphor and enabled users
to design the hand interactions on their own. While these works mainly focus on raw sensor
inputs, we hope to make an authoring system that can
allow users to map their complex and versatile gestures with
plentiful behaviors of the AR contents through programming by demonstration.
Before diving into the authoring workflow, we first need to learn what types of gestures
are used in AR. We follow an elicitation study done in 2013,
which categorized the gestures in AR from 6 dimensions.
The first four dimensions describe the semantic meaning and spatial properties of the gesture.
These dimensions can be inferred from the user’s demonstration.
So, a user does not have to tell the authoring system explicitly about this information.
The rest two dimensions are special. The form dimension describes the status of
the hands during an interaction. The hand gesture with a static form only includes
a specific status of the hands. There is no temporal information in it.
And the hand gesture with a dynamic form has a series of changing hand statuses.
The temporal changes of this gesture are important. The flow dimension describes when the virtual
contents respond. For a discrete flow, virtual content responds
right after the gesture, and for a continuous flow, the virtual content responds during
the gesture. Obviously, these two dimensions are about
the input and output of a freehand interaction. We need users to specify them because the
different combinations of Form and Flow can result in different interactions.
Static input plus discrete output makes static provoking interactions in which virtual contents
respond right after a static gesture is detected. A light bulb glows after a user performs a
holding gesture is shown here. Static input plus continuous output makes
manipulating interactions in which virtual contents follow the users’ hands while the
static gesture is kept. Here the user is holding a virtual cup with
a fist gesture. Dynamic input plus discrete output makes dynamic
provoking interactions in which virtual contents respond right after a dynamic gesture is detected.
Here is an example that the user breaks a soda can by performing a clenching fist gesture.
Dynamic input plus continuous output makes synchronous interactions in which virtual
contents respond synchronously to the movement of the hands during a dynamic gesture.
The spring here stretches following a pinching gesture.
We design GesturAR authoring interface so that users can make any of these interactions
by selecting an input object, and connecting it to an output object in AR.
Here I briefly show the authoring workflow of GesturAR.
First, the user creates virtual assets by either scanning, sketching, or importing existing
models. Then, the user demonstrates the gesture. Note
that he selects the type of hand gesture before starting the demonstration.
After that, the user selects an output behavior and connects it with the input gesture.
Finally, the user can try out the freehand interaction.
Here is another example for creating a dynamic-provoking interaction.
The user connects a patting gesture with a shrinking animation.
For synchronous gestures, we let users create the animation first, then demonstrate the
hand gesture using the animation as a reference. The animation is then mapped to the hand gesture.
Next, I will explain the detection algorithms used in our system.
We use the 10 hand joint angles to represent a static hand pose.
During detection, the input hand pose and the demonstrated hand pose are put into a
simple Siamese neural network together. The network then tells whether the two hand
poses belong to the same gesture. We train the Siamese network with 18 different
gestures to make sure it can effectively identify the hand poses.
To detect dynamic gestures, we first convert the gesture into a list of states.
Each state is described by a hand pose, the moving direction of the hand, and the direction
of the palm. As shown here, a waving hand and grab gesture
can be separated into 3 states. We assume the user is performing the same
dynamic gesture if the lists of states are matched.
For synchronous gestures, we extract a numeric key value from the gesture and map it to the
progress of the virtual content. Due to different hand poses, the key value
can be different. They can be a distance, an angle, or a diameter.
Here shows the 8 types of synchronous gestures support by our system. It is possible for
us to include more types in the future. To conclude this presentation, I show some
interactive AR contents that were created by our system.
Thank you for your time. I am happy to take any questions.