Hello, everyone. I am Arda Ege Unlu and I will be presenting PAIR, Phone as an Augmented Immersive Reality Controller.
We did this work at The University of British Columbia with Robert Xiao.
So immersive head mounted augmented reality allows users to overlay 3D digital content
on their own view of the world. Current-generation devices
Primarily support interactions through gesture, gaze and voice.
which are readily available but mostly lack precision and tactility,
rendering them fatiguing for extended interactions.
We also have controllers like Hololens Clicker, but these are also not very expressive.
We propose using smartphones which are also readily available
as companion devices complementing existing AR interaction modalities.
We leverage familiarity with smartphone interactions coupled with support for precise tactile touch input
to unlock a broad range of interactions and applications.
For example, here
you see an AR rendered sword and a bow, we have some other applications you will see in a second.
Smartphones have been previously proposed as companion
devices to AR headsets as controllers, notably most of these past
systems have employed external tracking systems to obtain the 6DOF - spatial - position of the phone.
So our goal with PAIR
is to demonstrate the feasibility of using an off-the-shelf phone
together with an off-the-shelf unmodified headset.
In terms of implementation,
in order to use
the phone as a 6DOF controller, the phone and the headset needs to be tracked in a uniform coordinate system.
We accomplish this by having the phone and the headset track themselves
in their own coordinate systems.
and then we align these coordinate systems by displaying these grids of markers around the bezel of the phone
and then detecting them with the headset's front camera.
This provides the phone's 6DOF position in the coordinate system of the headset.
and at startup time
This initial detection of the grid can provide
the initial alignment. Also by using a bezel of markers
we gain some amount of occlusion resilience.
Here you see some of the bezel configurations that we have
used for
different applications.
If the visual-inertial odometry we used to track the phone
was perfect, the markers wouldn't be needed after the initial alignment
however the phone's own tracking is error-prone and susceptible to drift over time
especially after rapid movements or as the phone moves into
unexplored places within the environment
For this reason PAIR doesn't rely on the initial alignment alone but continuously
scans for markers. And whenever markers become visible, we realign.
This ensures consistent and healthy tracking.
Yes - so what kind of interactions do we enable?
Phone can be an input device
and an output device - obviously
we have a display, tactile feedback, audio output
And now we also enabled 6DOF input and touchscreen input.
and there are curious ways to combine
these two - 6DOF and touchscreen input.
We provide two novelties that we name the 3D trackpad and the catapult swipe.
yes
This is a room level
fruit ninja application.
Here, you see the 6DOF
tracking a stable and fast enough to enable
a room level fruit ninja app like this.
Yes - here the user is browsing a catalog of furniture
and the furnitures are popping up on top of the phone.
And now, by using the familiar touchscreen interactions
like, panning a finger, zooming in, rotating, using a color palette
the user can customize this furniture.
And what is this? This is our three-dimensional
virtual trackpad, and unlike a regular trackpad
that tracks a finger on a fixed 2D surface
here the phone's orientation determines the plane of motion.
So that plane of motion, you see, as that grey plain.
And the user pans the finger to move
that cube in a three dimensional volume.
Here the position and orientation of the phone
together with swipes
can be used to initialize objects into the AR world.
Swiping dollar bills - into existence.
Yes, you see the user kneeling on the floor.
There's a quiver there with explosive arrows,
he picked them up. (Blows up targets.)
And evaluation - We tested the 3D positional accuracy
of our tracking approach in a Vicon equipped lab.
We recorded 35 different gestures and activities, activities like idling, walking,
walking and looking around
and gestures are stuff like
vertical swipes, in and out swipes, horizontal swipes under different variations.
marker variations correspond to different marker bezels.
that you have seen on the previous slides
like a full bezel, just the left column, just the top row, stuff like that
speed is either fast or slow
and orientation corresponds to, for example,
(gestures) for horizontal swipes whether the phone is facing upwards, leftwards, rightwards, and so on.
let's take a step back
and think about what kind of errors we have and what are the sources of error we have?
Hololens tracks itself, which is error-prone
the phone tracks itself
in our case with ARKit, error-prone
and then, we display markers on the phone,
and detect them, we use Aruco, error-prone.
So three sources of error.
In our approach we correct for the phone's own tracking by using Aruco detection.
We could have used the perfect system
instead of Aruco, or as close to perfect as possible, like Vicon,
that we named "the baseline (optimal) ARKit error".
In which we correct ARKit errors with Vicon reported phone positions.
and in ARKit-only error(2), we only do the initial alignment using Vicon
then do not correct at all after that.
So the error accumulates.
In the third one, the final error, this is our system's error.
We correct the ARKit error
using Aruco tags, Aruco corrected positions.
So here we see that final errors (3) are significanly lower than
ARKit-only errors (2), which is good news.
We also found the speed to be significant.
Which implies that faster motions are producing more error
suggesting latency
plays a part here.
Also marker position and orientation are not found to be significant.
Implying different configurations can be used confidently
for a healthy tracking experience.
Yes. In terms of limitations we have some 150 Ms of latency.
This comes through
a combination ARKit, networking, marker detection
all contributing to it.
These all happen in parallel but still add up.
Sometimes this produces visible lag.
in terms of drift: rapid motions
produce motion blur on camera, which comes into play
in monocular slam, making it into just inertial odometry.
For mitigation of drift and latency
different approaches can be taken:
new LIDAR enabled phones can be used, a more optimized
tracking algorithm can be used
communication be reduced by using something like BLE
also, just different
types of synchronization
can be used by using the historical head positions, rather than the most current one we use in our approach.
In conclusion, we demonstrated
the viability of the smartphone as a 6DOF controller for AR.
And the versatility and expressiveness of familiar smartphone interactions.
in this context.
Yes. Thank you for listening.