Freehand gesture is an important input modality of AR applications.
However, in most prior works, end-users can only use a few gestures with pre-defined functions,
and have difficulties creating customized freehand interactions in AR.
We propose GesturAR, an immersive authoring environment for creating freehand AR applications.
Users can customize hand gestures through embodied demonstration,
complete the application authoring by in-situ visual programming, and try it out in real-time.
Now we will explain the workflow of GesturAR by creating an interactive cookie monster in AR.
The first step is to create virtual assets that will be used in the AR application.
In Creation mode, GesturAR allows users to 3D-scan the surrounding objects.
Users can also sketch in mid-air or import CAD models.
Next, the user wants to hold the cookie monster using both hands.ﾠ
The user enters the Authoring mode, and demonstrates a 'holding' gesture using the
cookie monster as spatial reference. ﾠ The gesture is visualized as a skeleton model,
and the user can adjust the gesturing space by manipulating the bounding box of the model.ﾠ
To create an interaction in AR, GesturAR adopts a trigger action programming metaphor.
Here, the user selects a 'following' action from the menu,
and connects it with the holding gesture.
Now, the user can hold the cookie monster using the same holding gesture.
The user also wants to squash the monster by patting its head.
First, the user creates the 'squashing' animation of the monster by changing its scale.
Then, the user demonstrates a 'patting' gesture and connects it to the icon of the animation.
Users can test the AR application in Play mode. GesturAR keeps track of the user's hands and
compares them with the demonstrated examples. Here, the corresponding action of the cookie
monster is triggered when a similar gesture is detected.
We showcase a rich collection of ARﾠ applications created by GesturAR.
GesturAR enables users to interact with AR objects in multiple customized ways.ﾠ
Users can define their own ways to interact with AR objects.
Users can also create AR objects that can be manipulated in a similar way as real objects.
Furthermore, users can synchronize the reactions of the AR objects with their hand gestures.
In this example, users can use GesturAR to prototype virtualﾠﾠ
avatar and robot agents that can respond to customized hand gestures.
With GesturAR, users canﾠ enhance their conversation
by incorporating hand gestures with 3D content in animations.
Here, a user explains the Heliocentric model to another user.
Users can also build room-level AR games using the in-situ visual programming interface of GesturAR.
Here, a user is playing an AR treasure hunt game. First, the user restores the broken treasure map.
Then, the AR clues that appear leadﾠ them to the key and the treasure box.ﾠ
Finally, the gesture of opening the correct box triggers the winning animation.
We conducted a three-session user study to evaluate the
performance of our system with 12 users. In the first session, the users created andﾠ
tested multiple hand gestures to evaluate the accuracy of our gesture detection algorithm.ﾠ
In the second session, we evaluated the embodied authoring interface of GesturAR.
The users authored multiple hand-object interactions and manipulated those objects
using the authored hand gestures. In the last session, we evaluated
the overall usability of our system through a table-top interactive AR game.
Thank you for watching and please refer to our paper for more details.