Thank you all for having me here.
Today I'm going to present our smartphone prototype, Free-hand
Manipulation Portal-ble.
On the screen is a roadmap of today's presentation.
I will now start with our motivation.
As we all might have somewhat played augmented reality
applications on our phones, these apps
are not a stranger to us.
Here on the left is an IKEA place app,
where a user places a virtual chair in the room.
Another example on the right is a popular game, Pokemon Go,
where users interact with a virtual Pokemon
by swiping up the screen, and to throw a ball.
Notice that these interactions are all
happening in a 2D screen.
However, the virtual contents were
meant to be integrated as a part of a 3D environment.
This junction prompts us to wonder,
what could be an alternative way to interact with AR objects?
Prior work indicates that free-hand manipulation
can be a candidate.
Here I'm showing a sample of people
using free-hand manipulation to directly change
the scale of AR objects and move their locations
in head-mounted display settings.
Prior work also looked at free-hand manipulation
in smartphones to explore the tracking issues, performance
relative to screen, while user is
in a mostly immobile situation, and perception
issues in smartphone AR without free hand manipulation.
We want to explore what it takes to make free-hand manipulation
more practical on smartphones, and creating an intuitive
experience, especially when the user is fully mobile,
and how can we accommodate for those issues?
For this purpose, we need to create a working prototype
and gather first-hand knowledge.
What does it mean to be fully mobile
with free-hand manipulation?
In real life, objects can be within reach, or the red region
animated on the screen right now,
or beyond user's immediate reach.
It is animated in the blue right now.
Objects beyond reach are inactive,
but can be made active when a user goes nearby.
This mobility means users should be
able to use free-hand manipulation
without any restrictions to a fixed location.
We need to also account for adjusting
the imprecisions of grabbing virtual objects,
and change this mixed reality physics accordingly.
Image on the left shows a hand grabbing a real object, where
a user gets plenty of feedback from the haptics, texture,
and the material.
In AR, these are non-existent, making
it easy for users to overshoot, as the figure on the right
indicates.
Further, rendering virtual objects with correct lighting,
shading, and shadow helps to understand their relationship
to the user.
We can see the figure on the right gives more
hint of perspective.
Finally, we must choose the right type of hand tracking
that is stable--
robust for practical tasks.
We opt for depth-based camera for the relative stability
of their complex color backgrounds and lighting
conditions.
So with all these considerations in mind,
we implemented our first prototype
with a leap motion attached to a smartphone.
A leap motion device has a dual camera
that detects hand position up to 120 Hertz.
Because leap motion does not work on a smartphone,
we connect it to an Intel computer stick
that runs our script to capture data using interface Orion 4.0.
To ensure portability, the computer stick
is powered by a battery and mounted onto a portable case.
We then deploy web sockets of a client structure
to allow for real time communication.
The smartphone runs a unity scene with AR core
to localize its position in AR environment
and compute the incoming hand data for interacting
with virtual objects.
We're now moving forward to describe the gesture
recognition.
We train a model to predict five different gestures using
motion invariant local projections of fingertips.
The slide here shows an example of such a feature.
When our user's hand rotates, the feature value,
or the red line here, does not change.
It only changes when the finger crosses in.
We train 30 of these features for 16,000 samples,
and we're able to achieve over 98% of accuracy
for predicting the idle gesture, the palm
gesture, the pens gesture, the fist gesture, and a pinch
gesture.
Since most leap motion cameras and smartphones
RGB cameras' intrinsic metrics are automatically
calibrated by their unity assets,
we only need to care about extrinsic camera calibration.
The video on the right shows a user
with our interactive script to adjust
the position of the opaque white virtual hands to overlap.
The skeleton color here is deliberately
made obvious for understanding the calibration process.
In real time, it can be transparent or hidden.
So far we went over the main parts of the initial prototype.
Remember, we create this prototype
for identifying practical issues specific to fully mobile AR
experience on smartphones.
Inspired by experience prototyping protocol,
we designed two guidelines and a series of instructions
to allow us to get first-account knowledge.
We recruited 12 students from a research seminar,
and each of them designed a content, creation,
and a content manipulation task for their own participants.
This way we have a wide variety of tasks to understand
the potential practical issues.
After analyzing the reports, we found
that with free-hand manipulation on smartphones,
depth-perception issues affected users in two ways.
First, user hesitated to a distant object,
as indicated on the video right now.
Then, after they moved to the virtual objects,
these objects are still hard to grab.
The video shows how a user struggles
to find the right depth to the red virtual ball.
Notice the user tried different grabbing strategies,
but was unable to pick up the ball.
Even at the right depths we noticed manipulation issues.
Here on the right, a figure demonstrates
what happens in the recessed grabbing
when the user's hands reach its collision detection area,
but eventually fell out when the finger closes.
Lastly, we found participants in general
are unaware of the tracking limitations,
and their hands can be outside of sensing
areas, which causes a failed manipulation experience.
To mitigate these issues, we introduce
four feedback mechanisms and an improved manipulation design.
The first feedback is a progress wheel
designed to mitigate spatial location issues.
As a user approaches virtual objects,
the indicator on the object completes,
and it turns to blue when the object is within reach.
We designed a sound-pitch based type mechanism
that uses a raycasting-like technique
to select virtual objects and emit sound.
The technical details can be found in the paper.
[SOUND EFFECT]
For dealing with objects within reach,
we design a highlight and haptic feedback.
The highlight feedback helps to distinguish interaction states.
When a user's hand reaches a virtual object, it turns green.
When the user grabs it, it turns blue.
Similarly, when a user's hand reaches the object,
the phone slightly vibrates and gives a hint of feedback.
In order to mitigate manipulation issues,
we redesigned picking up conditions.
When a hand is near the object, the detection region
expands to the center of the palm
whenever at least two fingers are inserted.
The blue dots here are for visualization purpose.
They are not actually visible.
So when the pinch is complete, the detection region shrinks.
We can see how this technique helps to mitigate
recessed wrapping issues.
Occasional tracking errors and behavior noises
are filtered with a moving window filter,
so that accidental dropping of virtual objects are reduced.
In the video, you can see the finger temporarily in and out
of the detection area, as affected by tracking errors.
And with the future, we can get more consistent predictions.
Here on a screen we showed a comparison
with and without manipulation improvements--
on the left, a naive implementation,
where a virtual object accidentally dropped
with rotational movements, and lost due to recessed grabbing.
On the right, you can see the object stays in the hands
while the hand moves and rotates.
Finally, we evaluated our feedback and manipulation
design.
The study contends with two tasks.
The first task is picking up a distant virtual object.
This task has two sub-tasks-- one where the virtual objects
stays without moving--
one where object moves randomly at 20 centimeters per second.
We also recorded a case where a participant picks up
a physical distant object.
Task 2 is the assembly of AR blocks
into a virtual playhouse.
We recruited 12 participants with an average age of 23.
Here's a video showing what happens in the motionless task.
The baseline is an naive implementation.
The baseline video here shows no feedback and manipulation
redesigns were added.
On the right is a participant using Portal-ble
with visual feedback.
The video that plays now demonstrates
a baseline condition where a user
is trying to pick up a moving object without any feedback.
Here is the condition with visual feedback.
Here's an example of task two for building virtual houses.
Participants need to assemble a virtual house
from four pillars and one roof generated
at random orientation.
Participants need to align the blocks
and rotate them with extra care.
We show our results here.
Portal-ble's conditions are noticeably faster
than the naive implementation.
When a user approaches a moving target,
the shaded area here indicates average standard error
for all participants.
Our results show Portal-ble helps participants
to reduce hesitation towards a moving distant virtual object.
Portal-ble allows users to be more
successful in grabbing virtual objects compared
to naive implementation.
The percent on x-axis indicates success rate,
and the y-axis contains combined conditions.
For motionless tasks, Portal-ble is 20% more successful
than the baseline, and the success rate
is almost more than doubled in Portal-ble's moving conditions.
And the error bar here indicates standard error.
We rate scores for completion of AR houses.
Portal-ble has an average score of 0.72,
while baseline condition amounts to 0.48.
A paired t-test comparison shows that the difference
is significant with p values equal to 0.01.
We compared the preference and helpfulness between baseline
and each individual Portal-ble's feedback with a 5-point likert
scale.
We noticed that the baseline is neither preferred
nor seems to be helpful compared to the Portal-ble's condition.
Sound is less preferable, but still helpful
compared to the baseline.
We measured the overall cognitive load
with NASA-TLX, which is a self-reporting form originally
used by NASA scientists to measure cognitive load,
such as mental and physical demands.
The x-axis here are the different categories in TLX 4.
And the y-axis here are the raw scores--
the lower the better.
On this figure, we can see that Portal-ble's cognitive load
is significantly less than a baseline in every measure.
So the takeaway here is that we built a prototype
to list practical issues when combining
free-hand manipulation with mobility on smartphones.
And we made a set of visual and haptic feedback designed
to mitigate perception issues, and manipulation
redesign to improve accuracy, and empirical studies
supporting Portal-ble's redesign mechanism in terms
of mitigating issues mentioned above.
We aim to raise popularity and research
interest for free-hand manipulation
on smartphones and other handheld platforms,
and thus open-sourcing our project
at portalble.cs.brown.edu.
And that is all my presentation, and thank you all.
Right, good.
We've got time for questions, so ask away.
[SIDE CONVERSATION]
Thanks for your talk.
I was wondering when you did the user study, what
were user comments on the haptic feedback,
since you get feedback on the right hand that's
holding the phone, but you're interacting with the other hand
that's in air, sir?
Yes, there's actually a term called haptic retargeting,
which-- sometimes you don't need haptic on the hands
as long as you have visual feedback that goes along
with it.
So the users actually commented very positive for the haptics.
They think-- I think this slide shows--
many of them think it's very useful, like mostly,
because otherwise, AR objects are--
you don't have any clues when you reach it or not.
Even though it's not a direct feedback,
it still gives a hint.
[SIDE CONVERSATION]
I guess I'll ask a question, actually.
So I'm kind of just curious if you could comment
on some of the long term ergonomics of this kind
of project, because we all know that holding that in mid-air
is tiring.
So you did some studies for a shorter period of time.
But did you have any insights about longer periods?
Yeah.
It's actually a very good question,
because the AR interaction, especially
with free-hand manipulation, is known to have a degree of harm,
in fact, where people's arms become really sore.
So we actually had an artist come in
and try to use our system to make a pen in 3D.
That lasted for an hour.
So the artist was able to complete the drawing
she was trying to do, but she did report
some soreness with the button.
Nevertheless, she was able to complete the task.
Very impressive.
Thanks.