Hello everyone. My name is Feiyu.
Today it is my pleasure to be here and present our paper.
I would like to thank my mentor - Yan,
and collaborators from Meta Reality Labs Research for their guidance and support on this work.
Augmented reality (AR) head-worn displays have the great potential to support everyday information needs in the future.
In current AR operating systems, virtual content defaults to be world-anchored.
They stay at a fixed location until being manually moved by the users.
Such setup offers limited mobility to the AR content.
Users often lose complete access of their virtual applications after moving to a new space.
There are three common solutions to transition the AR user interface (UI) applications:
(1) Tag-along, in which virtual content loosely stays in the user's field of view and follows them around;
(2) Drag and Drop, in which users manually grab and move the virtual content to a new location;
and (3) re-instantiate, in which users reopen and place a target app at a new space via a menu.
There are obvious limitations to these methods.
For example, the AR content would become intrusive and obtrusive while following the users around,
or they would consume continuous input and attention from the users during manual interactons.
Inspired by current mobile phone interfaces,
a better soluton could be that we offload different levels of user control to the system.
For example, if we offload some user control to the system,
similar to the Siri suggestion feature on iOS, the system could suggest some possibly-needed apps,
and the user holds the final control over which app to open.
If we transfer the user control entirely to the system,
the system would surface the predicted app in front of the users automatically without any input needed.
In this research, we are interested in learning about
how combining different levels of automation and user control could help transition AR UIs on-the-go.
To understand the challenges with current UI transition mechanisms,
we conducted a bodystorming design workshop.
We recruited five participants from the industry with rich experiences in AR/Virtual Reality (VR)/Mixed Reality(MR).
We built a prototype application on the Microsoft HoloLens 2.
In the prototype, participants had access to eight widget UIs to assist their daily tasks,
and we implemented the three existing solutions mentioned earlier to transition these UIs to different locations.
The workshop was conducted remotely.
Participants were asked to install the application on their HoloLens 2 devices,
and conduct a sequence of tasks in their own home environment.
The tasks required them to move between three indoor spaces:
the home office, the kitchen, and the living room.
In each space, they needed access to at least one AR widget UI to assist their real-world tasks.
Participants were asked to identify painpoints during the tasks,
and brainstorm about potential solutions to these challenges in an online co-design workshop.
Our results highlighted three common challenges.
First, placements of the widget UIs,
Participants complained that the placements and organizations took a lot of unnecessary efforts.
Second, awareness and recall.
Participants mentioned that they often forgot why and where an app was placed beforehand.
And last, high-effort acquisition, in which participants wish they did not have to reopen an app
manually just to look at some simple information.
In the design workshop later, three solutions stood out and received the most votes from the workshop participants,
which are (1) Wrist-based glabceable UIs, (2) Snap to planes or objects,
and (3) Everything in the right place at the right time.
After figuring out the major challenges and potential solutions,
in the second study, we developed three interfaces to help transition the AR UIs
with different levels of automation and controllability
Due to the pandemic and to avoid unstable multi-room tracking with current AR systems,
we developed a VR-simulated AR environment for evaluation.
The environment included three rooms:
a home office, a kitchen, and a living room.
Participants could move between these rooms by touching a button on the virtual door,
so we were able to reuse the same walking area in the physical world
to simulate the transition procedures and interactions in different spaces in the virtual environment.
In this task, participants were asked to imagine that they were the owner of this virtual home.
They interact with various objects in daily tasks,
while wanting to know some information in an AR application.
For example, they want to know what to pick up next for a recipe after opening up the fridge.
Thus, a multiple-choice question pops out near the object simulating their thoughts of mind,
and participants had to check an AR app to answer the question as fast as possible while prioritizing accuracy.
As shown in the video on the right, in a single trial,
participants follow the virtual instruction board, go to a different room, use a object,
and access an AR widget to answer the question popped out in front of them.
Inspired by the solutions in the design workshop, we developed three interfaces
with different levels of automation and controllability.
First, Wristpack, in which after users enter a new space,
all AR apps would be attached to the user's wrist automatically.
Users could open an app by grabbing it out of their wrist.
The second solution was called Semi-Auto.
In this condition, the system would suggest three most relevant apps on the user’s wrist when a question was asked,
and the user could pick which app to open and when to place it.
And the third solution was called Fully-Auto.
As its name implies, the system would automatically place the widget it thinks the user might need without any input from the user,
and the user cannot interfere with the system's choices
Last, we also included the drag/drop and tag-along interfaces as a baseline for comparisons.
In this condition, users would always have to recall where they left a widget and retrieve it manually,
simulating existing solutions in current AR operating systems.
To learn more about user's perceptions towards the prediction errors,
we simulated imperfect prediction accuracy.
In both Semi-Auto and Fully-Auto, the system would suggest the wrong application 25% of the time.
However, since user had no control in the Fully-Auto condition,
a higher cost was induced to recovery from the errors.
We conducted a fully remote and unsupervised user study using the dscout platform
with 40 regular Meta Quest 2 users.
We collected both objective and subjective measures for evaluations.
Here shows a summary of the key findings from our second study.
First, we found that Semi-Auto performed the best both subjectively and objectively.
Second, we found that Baseline performed the worst both subjectively and objectively.
Third, we found that Fully-Auto led to lower level of agency compared to the more manual conditions.
Fourth, we found that Semi-Auto led to an equal or higher level of agency compared to the more manual conditions.
Last, we found that the sense of control and ease of error-recovery
were the determination factors for user experience.
Our results highlighted three lessons learned.
First, current solutions are not good enough to transition AR UIs,
and we need to develop ultra-low-friction interfaces on-the-go
when users have limited cognitive bandwidth available.
Second, similar to the Semi-Auto condition, we need to combine and balance automation with controllability
to create the best usability outcomes without agency being compromised.
Last, we need to research ways of failing gracefully with back-up plans.
to enable users to recover from the prediction errors.
That is the end of the presentation. Thank you for listening.
Please refer to our paper for more details.