Hi, everyone, I am Fengming He.
Today, I am going to present our work, Ubi Edge: Authoring Edge-Based Opportunistic Tangible
User Interfaces in Augmented Reality.
This work was completed with my collaborators, Xiyun, Jingyu, Xun, Tianyi, and my advisor
Dr. Karthik Ramani from Convergence Design Lab at Purdue University.
Let’s get started.
Tangible user interfaces allow a user to interact with digital information through the physical
environment.
With the invention of mixed reality, TUI has become one essential way for rendering haptic
feedback in AR.
Further, opportunistic TUI tightly maps the affordance of physical objects with the functions
of virtual widgets like buttons, sliders, and knobs.
The tight mapping of opportunistic TUI provides a more intuitive connection between the user
inputs and digital services.
However, such strict mappings usually require users to find an object with a similar shape
and size as the AR content,
which limits the generalizability of TUI.
And this inspires us of leveraging local geometric features, instead of the entire object to
customize opportunistic TUI.
Specifically, we focus on geometric edges and sides which are ubiquitous on everyday
objects.
Edges can also provide sharp tactile feedback and users do not need to pay full attention
when interacting with edges.
For example, we know where the edge of a desk is through direct touches, and we do not have
to look at it.
Then how to let users utilize edges to author an opportunistic TUI application?
We divide the input edges and digital output into different categories.
And different combinations of input and output can result in various TUI applications.
Specifically, we categorize the touch input into binary touch and consecutive sliding
so that the system will treat the edges as buttons and sliders respectively.
We also divide the output into discrete and continuous.
For example, in the subfigure b with a combination of consecutive input and continuous output,
the cube’s movement will be linearly mapped to the user’s sliding input.
After understanding our system’s basic taxonomy, an important problem now is how to track the
geometric edge of an object.
In Ubi Edge, we design a vision-based processing workflow so that our system can accurately
detect the edges on everyday objects.
Rather than extracting all edge lines on the object, the edge detection here is to extract
feature points on the object.
Then we let users define their own tangible edge inputs by sliding their fingertips on
the overlaid feature points.
In detail, we first used a real-time RGB-D edge detection algorithm to get geometric
edge points.
We also use a network-based keypoint descriptor to track the 6DoF of the object.
Here are some edge point detection results using our tracking workflow.
Now we use an example to demonstrate the workflow of Ubi Edge.
One day, a user wants to author a smart light controller TUI by interacting with the frequently
used cup.
For example, the user wants to turn on or off the smart light by touching the handle
of the cup.
The user first enters Scan Mode of Ubi Edge and looks at the target cup for a brief moment.
Then our system captures the RGBD information of the object to extract geometric edge points
on the object.
Next, the user enters Author Mode to define the edge-of-interest by simply using index
finger to slide along the handle.
Now, the user sets the edge input type as binary so that our system treats the input
as a button as we mentioned earlier .
Then the user selects the ‘on/off’ digital function of the smart light and connects the
selected edge input with the digital function through visual programming.
If the user wants to set a continuous output, the user needs to specify the range of output
function.
Here, the user sets the cup rim to adjust the color of the smart light.
Now the user can turn on or off the light bulb using the handle of the cup.
Or the user can adjust the light color by sliding left and right along the rim of the
cup.
As the edge properties affect the edge detection results,
we evaluated the effectiveness of edge detection with different parameters including edge length,
sharpness, and distance between edges and the camera.
Here we used different rounded edge radius , and angles between two surfaces to represent
the impact of sharpness.
Overall, our results show state-of-art performance compared with previous works.
Here we showcase four TUI applications created by our system.
With Ubi Edges, a user can turn physical objects into a multi-function controller.
The users use the book to turn on/off the AR fan, let the fan oscillate and change the
fan’s speed.
Users can also use edges on physical objects to control IoT items.
The user here uses a scale to let the audio player start to play, jump to the next song
and change the volume.
With our system, users are able to create tangible AR games that use everyday objects
as controllers.
Here a user utilizes the game console’s ridge to interact with a basketball shooting
game.
Ubi Edge can also improve users’ task efficiency by enabling users to control tutorial-based
TUIs.
We conducted a two-session user study to evaluate the accuracy of the touch interaction with
edges, TUI authoring feasibility and overall system usability.
We provided 6 daily objects and they vary in size, edge numbers, and edge curvatures.
The first session is the quantitative evaluation of the touch interaction and the second session
is qualitative evaluation.
In the first session, for each object, users can choose any two tangible edges they want,
one authored edge was set to binary and the other edge was set to consecutive.
We asked users to perform five touches for each binary edge and slide along the consecutive
edge to reach specific points on the edge.
For binary edges, we counted whether the system detected touches.
And for consecutive edges, we recorded the completed time.
Overall, our system could accurately detect the touches and accurately manipulate the
authored tangible edges.
In the second session, we evaluated the performance of Ubi Edge authoring interface.
We asked users to finish 6 TUI authoring tasks which include all four combinations of inputs
and outputs we mentioned earlier in the taxonomy.
In general, users showed a high preference for using geometric edges as tangible inputs
and they agreed that repurposing tangible edges into controllers is intuitive.
Thank you for watching.
For more details, please refer to our paper and video.