- Alright. Thank you for the introduction.
My name is Han Joo Chae
and I'm a PHD candidate at Seoul National University.
Today I'll be talking about
Wall-based Space Manipulation Technique.
This work was done with my colleague Jeong-in Hwang,
who were also at the Seoul National University
and as part of the HCI lab working
with professor Jimwook Seo.
So I'll jump right into important questions,
now a days there are a lot of implications so...
How do we interact with virtual objects in AR?
I guess we can use hand gestures
or maybe we can use external controllers as we do in VR,
and they work pretty well in most cases.
Then how do we interact with distant virtual objects in AR?
This is a question that our research is trying to focus.
In particular we are interested in moving
near by objects far away and bringing far objects closer.
So to make it easier to understand let's say
we want to move a near-by soda can,
which is virtual by the way,
to a far corner of another table.
So the most intuitive way would be grabbing the can,
and then walking towards the table
as we do with real objects.
However, if we don't want to move because...
Including myself we're all lazy so we can use
a remote manipulation technique.
Here's an example of using a
hand center manipulation technique.
As we have just observed,
existing remote manipulation techniques are not effective
for moving objects forward and backward.
We often need to repeat same gesture multiple times
and selecting smaller and far away objects can be difficult,
and also in manipulating far away objects,
could easily become overly sensitive for fine-grain control.
Also due to the limitation of human perception,
recognizing exact positions of virtual objects
is more difficult than those of the real ones,
and it gets even worse when they are farther away.
For example, here's...
We intentionally placed a soda can off the table
using the Microsoft HoloLens, however, from far away,
it seems like this can is right on the table
and we don't want this to happen.
There have been a lot of previous work
on 3D remote manipulation and I would like to
briefly touch on some of them.
There are closely related to our work.
First is a HOMER technique introduced by Bowman and Hodges.
It is one of the most popular technique used in VR and AR
and it supports remote manipulation by moving,
by mapping the object to user distance
on to the initial user to hand distance.
However, it is still difficult to control
far away objects due to the increased sensitivity
and the limitation of human perception.
World Scaling such as, World in-miniature
and Scaled-world Grab iso tropically scales down the world
for manipulation in VR.
However, they often make the world too small
for fine-gran control or do not
sufficiently scale down the world to be effective.
Adding handy snapping functionality by detecting edges
and surfaces in AR has been introduced
as in SnapTolReality and Projective Windows.
Snapping does not help the user to place objects
on arbitrary locations where
no real constraints are available.
Our goal is to develop a new technique to efficiently
move virtual objects forward and backward
regardless of their distance to the user.
At the beginning of this research my college Jeong-in
and I were just sitting in the lab and brainstorming
and I said "hey, why don't we bring the wall over there,
"to us and then do whatever we wanna do
"and just place ably put it back."
Of course it was a stupid idea, because first,
it is impossible, to move that large wall,
and second, even if we could,
if I move that wall then probably it will collapse
the building and we don't wanna jeopardize that.
This came to our minds.
This clip is from the movie Dr.Strange
where his astral body comes out from his real body
and somehow, affects the real world.
Similarly our idea was to take out an astral wall
from the real wall and do whatever we wanna do with it,
and then put it back.
Here's how our wall based space manipulation,
in short WSM technique works.
Let's say our user Alice is in the room,
where our virtual cattle and the virtual wall frame in red
are placed on the table and the wall respectively.
When she scans the room, astro-walls,
which we call wall plains,
are created and overlaid on top of each detective walls.
Here we have two walls detected,
one on the left and one behind the table.
Also mash lines are drawn on the surface
of the detected real objects,
such as a table and the shelf.
Alice wants to move the kettle
to the closer edge of the table.
She first drags the wall plane in the front,
to squeeze a space between herself and the wall.
All the wall planes, virtual objects
and surface measures of the real objects
are moved and scaled accordingly.
Then in the reduced space, she moves the kettle
to the closer corner, on the surface mesh of the table.
As the space is reduced,
she only needs to move for a small distance.
When she double taps the wall plane,
the space returns to normal
and all the objects are currently placed as intended.
Now I'm going to show a quick demo video
of our working prototype developed on the HoloLens.
In this prototype we added WSM
to the direct HOMAR technique.
So when the user scans the room, the wall plans are created.
The user drags the wall plain to squeeze
the space, so that she can move the virtual kettle
and the soda can...
with smaller motion.
Double tapping the wall plane,
brings the space to initial...
Bring it back to the initial position.
This is an example of using multiple wall planes
at the same time to scale the space in multiple dimensions.
For more detail and information about the implementation,
please refer to our paper.
Now I'd like to talk a little bit about the decisions
that we made during the design process.
First things first. Why Walls?
Most indoor places are surrounded by rectangular walls.
They're large and flat surfaces are easy to detect
and robust from occlusion.
By the way, these photo's were found from the website
of 2017 UIST, and if you look around
and sense there are a lot of people in this room
we can still identify, walls surrounding us pretty easily.
In our technique our role-playing
squeezes a space by one dimension.
Technically it is squeezed along the
surface normal of the wall plane,
but our intention is to reduce the depth of the scene
from the user's perspective.
This is particularity advantageous because,
from the user's point of view,
objects would look mostly the same,
but only a bit thinner.
In addition, by allowing the user to scale the space
using multiple wall planes at the same time,
we leave it to to the user to decide which
and how many dimensions they want to scale
depending on their needs.
Lastly, it is important that the user visually understands
how much the space is reduced.
It recognizes which location in the reduced space,
corresponds to which location in the real space.
When using the WSM, the user can understand
the reduced space by observing changes
in the surrounding wall planes of virtual objects
and surface measures.
We also drew grid lines on the wall planes
to better instruct the user about the changes in the space.
Furthermore, the surface measures can act
as reference points from the reduced space,
to the real space.
We conducted a user study
to verify the benefits of using WSM.
The top photo is the actual study set-up
and the bottom is a screenshot of what the participants
saw during the experiment.
We used three meter by three meter wall,
located 11 meters away from the participants
and gave them a simple task of docking the red source cube
to the green target cube.
The source and the target positions were randomly generated
with a fixed distance of eight meters along the Z axis
and the participants were only able to proceed to the next
trial when they successfully docked source to a target.
It was a within subject design experiment
and we recruited 12 participants.
Here we want to note that one participant
couldn't properly wear the HoloLens
and was only able to see about half of the screen
so we analyzed and reported the 11 participants results.
Independent variables were WSM states,
whether WSM was used or not.
Direction, whether moving near source to the far target,
or far source to a near target.
Three different cube sizes, small, medium and large.
We control other variables because
we mainly wanted to see the affect of WSM.
We measure performance time and travel distance and SUS.
Each participant performed 48 trials
and the total of 528 trails were recorded.
We analyze the results of the test completion time
and hand-traveled distance using three-way arlmanova
and pair wised comparison with one correction.
When WSM was used, the participants
completed the tasks significantly faster
and moved their hands significantly less.
When the directionals mirror too far,
the participants performed significantly worse,
with more hand movement.
The participants completed the task significantly faster
when the cube size was large.
We also analyzed SUS scores.
The average SUS score was better with WSM than without,
however, the difference was not significant.
There were also many interesting interaction,
that were statistically significant.
These three indicate that WSM as more consistent
and reliable, even when there were
changes in direction and cube sizes.
Blue line is when WSM was enabled
and orange line is when it was disabled.
These two indicate the moving small cube
from near to far without WSM was extremely difficult.
We believe there are two major reasons for this.
First the former technique is overly sensitive
when the objects are far away,
and second, it is difficult to recognize
exact positions of distant objects
due to the limitation of human perception.
Moving the cubes near to far without WSM
also showed interesting three-way interaction where,
the participants move their hands more,
even with large cubes.
We strongly believe such unreliable results
are the reason why we need to use our WSM technique,
especially when we move near objects to far away.
Now I'd like to discuss important insights that we had.
The results of the completion time,
and the hand traveled distance indicate that
when WSM is applied to the direct HOMER technique,
the user moved the objects forward
and backward more efficiently, with less arm fatigue.
Also the results of the significant interactions indicate
that the direct HOMER technique without WSM,
significantly suffered when the objects were small
or tasked was moving near by objects far away.
By contrast when WSM was applied the performance
was consistently high overall.
We were also able to gain feedback from the participants
about how they felt when they interacted
with the wall planes.
All participants well understood the concept of the
WSM and the wall planes helped them recognize the positions
and the distances of the objects within the reduced space.
WSM is intended to be flexible.
It let's the user to decide on which
and how many wall planes they wanna use
depending on their needs.
In addition, although we implemented our prototype
on the HoloLens,
WSM is not limited to the direct HOMER technique
or the HoloLens and it can be applied
to various techniques and devices, in a non-intrusive way.
In other words while the concept of WSM
can be used to manipulate space,
methods to point, select and drag
can be changed based on the application
and user scenarios.
Currently, if I talk about limitation...
Currently WSM only supports translation.
We believe that rotating objects in the squeezed space,
would also be very interesting,
so we plan to extend it to forward rotation
and scaling in the new future.
Complex surface measures were occasionally
difficult to be understood
and could potentially cause visual clutter.
Since these were out of the scope,
we did not put too much effort on this matter.
However, we believe simplifying
and selectively displaying measures
or removing unimportant real world objects
from the scene would be beneficial.
Our user study was simple and only use a single wall plane.
We plan to conduct another user study
which includes complex user scenarios,
and multiple wall planes at the same time.
Some insights might have been
too specific to the direct HOMER technique.
We home to imperially evaluate how our technique
could improve other techniques in the future.
Such as the indirect HOMER technique
which uses controllers or touch gestures on smartphones.
In summary, we presented that WSM
dynamically scales down the surrounding space,
for the efficient placement of virtual objects
in an indoor environment using augmented reality.
Our prototype extracts room boundaries to create
interactive wall planes and generates surface meshes
of real objects as reference points
from the reduced space to real space.
We discussed a set of design considerations
and insights that were unique
to the remote AR interaction.
Our user evaluation showed that WSM
performed consistently well
and significantly improved the efficiency
of the direct HOMER technique while
alleviating arm fatigue.
Thank you very much and I'll be more than happy
to take your questions from now.
(Applause)
- [Interviewer] Questions?
I'll start with one.
What happens in an environment where there are
no walls, like outside,
where the walls are not right angled to each other?
- I'm getting this question a lot.
I guess people are really interested in
open spaces, or when they are not really,
90 degrees to each other.
First of all we've focused on the walls because...
Not only because it's easy to detect,
secondly perceiving the change in space was so important
when you use this WSM technique,
and we found that users benefited from
the side walls as well, because we drew grid lines on those,
people were able to see how much
they were changing the space.
I think, the close environment is better for
this kid of a technique,
however, we still can think about landmarks
or other objects that can be detected
and use those to squeeze a certain amount of space,
but I guess we'll have to investigate
a little more about that
and see how people recognize the space when it's open
and there are no surrounding walls.
- [Interviewer] Walls is a good idea,
because their kind of ubiquitous anyways.
I would have...
Then another question which is...
You only do uniform scaling here.
Did you consider non-uniform scaling,
for example, scaling the space more that's for the weight
or it's, kind of that's closer to you,
in order to do the context that we're scaling?
- So now we let the user to decide on that much
Currently our development is just the arms.
We just took the average arm length so that when...
Which is 50 centimeters.
We mapped the distance from the user to the wall,
so that it can be scaled from at once,
so you can come all the way.
Thinking about this kind of a distance,
it would be a lot beneficial
to be using the contextual wire,
or non-linear scaling.
So, I guess I could investigate that in the future as well.
- [Interviewer] Alright then, let's thank the speaker again.