hello everyone i'm Jing Qian, today i'm presenting a talk on modality and depth
in touchless smartphone augmented reality interactions
touchless interactions such as hand tracking or dwell can be intuitive
entertaining and immersive to use. These interactions are
emerging on augmented and virtual reality
environments as supplementations to traditional joystick
or touchscreen. In practice interaction distances or depth
can affect touchless interactions. For head mounted displays
or hmds both the effect of modality and distances are well studied, but this
is not the case for smartphones. Why do we explore touchless interactions
on smartphones? as AR platforms, smartphones have much
larger user base than hmds. Thus exploring the effects
of modality and distance could potentially benefit
more users. Smartphone touchless interactions can be useful when a person
is cooking, or when a person is wearing a glove in
harsh or soiled environments that makes touch interactions difficult
or unsanitized to perform. Furthermore, the screen dwell interactions
on smartphones can also enable one-handed interaction
which can be useful in AR interactions. in this paper we want
to understand the fact of touchless modality
and interaction distance on smartphones and explore these three hypotheses.
we hypothesize that users would prefer dwell interactions over hand interactions
users would prefer close-range over
distant interactions the overall cognitive load would be
higher for close-range interactions. For detailed rationale behind these
hypotheses please refer to our paper
we built an AR application with iOS ARKit
and implemented dwell and hand tracking. The dwell interaction uses ARKit's
localization to find where the phone is pointing at
and the user can aim at different virtual objects
with a dot-like crosshair. when the user holds the phone still a
progress wheel rotates to complete. For hand tracking, we 3D-printed three
color markers to fit the index finger and the thumb to support an OpenCV based
color tracking we removed objects with similar colors
in the lab to avoid tracking issues. here are two example videos with our
system the left one demonstrates using hand
tracking to press on virtual pin buttons
the sphere on the index finger indicates where the finger is
on the right it is a video demonstrating the dwell interaction to press
pin buttons. For hand tracking a user can move their index finger over
the virtual element and the element will change to a
highlighted color when a user pinches it is considered
selected if the target is a button the selection
will automatically debunce if the target is a slider while selected
user can move the object to perform
translation and release fingers can deselect the object. For screen dwell
once the cursor aims at the virtual object a progress wheel
starts to refill. once filled, it is considered selected
if the target virtual object is a button it will automatically debounce
but when the object is a slider the object will
remain selected and move to where the cursor points to
in real time. Once the smartphone becomes still again
the progress wheel will refill. Once it's filled again, it is considered
deselect
we designed a 2x2 study to understand the effect of interaction distance and
modality here we test interaction in a close-range
where objects are within reach and in a
distance where interaction happens at about 2.4 meters
we recruited 15 participants with
average age around 29 years old and the study
was on average for 42 minutes most of participants have prior AR
application experiences and when asked about
familiarity with the concept of dwell interaction
all of them seem to understand the concept right away but this is not the
case for hand tracking. We designed our task
to reflect real life scenarios to get participants into the everyday mindset
of using the touchless touchless interaction
the two tasks here are pin-code task and the tv task
shown on the two figures on the right. In the pin codes task
a user needs to perform four selection tasks
or subtasks to match a randomly generated password
to unlock a virtual painting in a tv task the user was asked to perform two
selection subtasks and four scrubbing subtasks
to seek for specific moments in the video based on the experimenter's
commands which is demonstrated in the right picture here
we collected all these data and ran them through
a multi-level mixed-effect linear model and we found that distant interactions
are significantly faster than close-range ones. Specifically
screen dwell is significantly faster than hand tracking. No interaction
were found between modality and distance also we didn't find
any significant effect on the task ordering
so here we revisit our three hypotheses. We hypothesized that the overall
cognitive load would be higher for close range interactions
but we found that the overall cognitive load was not higher
for close range completion time. However we found a significant correlation
between overall completion time and the task load. The higher the
completion time the higher the overall cognitive load for participants the
figure on the right shows all the subcategories of NASA TLX
NASA TLX is commonly used score rating to measure
cognitive load we use the raw scores from 0-100 to
measure effort frustration mental demand
performance physical and temporal demand. If we look
at hand tracking only we notice that the
distant hand tracking is in general less frustrating less mental demanding
less temporal demand and also less performance rating
we do not see similar effect for screen dwell where distant
and close-range interactions have similar cognitive readings
for all the subcategories
we also hypothesized that users would prefer dwell interactions
over hand gestures and found that participants strongly prefer
screen dwell over hand tracking one possibility could be that the
familiarity plays a part in this as almost all participants were familiar
with the concept of screen dwell but only 17 were for hand tracking we
also noticed that hand tracking was not
natural for some participants as they exhibit
different pinch gesture than what the system
can recognize. Despite the low preference hand tracking is still considered by
some participants to be cool futuristic natural and fun to use
we further hypothesized that users would prefer close range over
distant interactions but the findings were reversed
this is a bit unexpected since AR contents are often situated
and meant to be treated like real objects and
we interact with real objects in a close-range manner
participants mentioned that they felt constrained
in hand tracking because their hands need to be in the smartphone's field all
the time as shown on the upper right
figure here where user's hand becomes not visible
on the screen if moved too far away to the right side of the smartphone
and for some people a natural forward pinch was engaged in hand tracking
but this results in the thumb occludes the index finger to the camera's
perspective resulting in issues with hand tracking
We also noticed that some depth finding issues can lead to
timeouts in close-range hand tracking in general from the system design
perspective it is better to inform that
user's what gesture should be used in the
beginning. Prior work already showed that users have good learnability to gestures
and can stick to those gestures once learned it
further suggests that components requiring higher
performance such as UI controls should be considered for distant
interactions lastly on the ergonomic side both hand tracking
and screen dwell has their shortcomings
participants mentioned arm soreness in both dell and
hand tracking situations for hand tracking a brief arm relaxation
can reduce the soreness as for screen dwell use both hands to
hold the phone almost eliminated further soreness
i would like to mention few limitations we have
firstly this study was framed in a context of pure touchless interaction
and not focusing on the difference or similarities
between a touch based interaction and a touchless interaction
The 3D translation and rotations were also not evaluated
but could be part of the future work the long-term benefits
or issues of touchless interactions were also
not within the scope of this paper
so here's a takeaway we did a study to compare
understands the effect of interaction distances
on hand tracking and dwell in a smartphone context
we find that overall distant interactions are faster
more preferred than the close-range interactions but for hand tracking
participants mentioned the close-range can be
futuristic and immersive when designing for AR applications
designers can consider distant interactions for UI components
or other interaction components that performance is higher priority
and in a fixed interaction distance the two modalities could be interchangeable
without extra cognitive cost thank you for your time
and that concludes my talk if you have any questions feel free to email me
thank you bye