- Introduction, can everyone hear me in the back?
Think so?
Okay.
So, I'm Benjamin Nuernberger.
I'm presenting our work on SnapToReality.
So, positioning virtual content
is important in many augmented reality applications
with respect to the real world.
So, for example, we have this virtual recipes app here
centered nicely on the cabinet doors in the back.
We also have this vacation to-do list application
positioned nicely near the corner of the refrigerator,
and similarly, we have the Maui weather app
positioned nicely near the corner of the table.
And this is just one example where precise positioning
of virtual content with respect to the real world
is important.
And in the physical world, we often take this for granted.
We can place things in the world
with respect to certain physical constraints
or design constraints.
Another scenario is in authoring
or creating virtual content.
So say you want to create a toy in this room,
let's say, Thomas the Train on this sofa.
And we want to move this virtual Thomas the Train
off the sofa onto a virtual bridge.
So, you might start by first building up
the columns of the bridge to the height of the sofa.
And you keep going, and we build out these columns.
And then you extend the bridge from the height
of the sofa onto these columns.
And then now, your Thomas the Train can easily
and seamlessly move from the sofa onto the bridge.
So, these are the kind of experiences
that we want to create.
So we asked ourselves this question,
"How can we precisely align virtual content
"with respect to the real world?"
given certain physical constraints
or certain design constraints.
And we quickly realized that an important tool
in many desktop 2D and 3D environments is snapping.
So snapping, you've probably all are familiar with it
in different applications such as PowerPoint.
Here you see PowerPoint allows you to snap
virtual content with respect to certain
geometrical properties of other content.
And of course, snapping has been researched many years.
And due to limitations on the time,
I'll just give you two examples.
Bier's work in the late 80s on snap-dragging
is one of the early works.
Today we can play with tools such as SketchUp
where you can extrude different content
to snap to the height of other content.
Now this is in the realm of snapping virtual content
to virtual constraints.
Our work is actually in the realm
of snapping virtual content to real constraints.
So in this realm, we have a line of work
with 2D image snapping, starting in the mid-90s
by Gleicher.
And more recently, Lee & Billinghurst applied this
in augmented reality
to an augmented reality drawing application.
Similarly, in the same year, in 2012,
Nobrega & Correia analyzed a single image,
extracted different image features
such as vanishing points and used those
to place virtual content in that single image.
Our work actually takes this a step further,
extracting these 3D constraints in real time
for 3D augmented reality snapping.
And I'd like to point out here
that snapping to virtual constraints
is inherently easier than snapping to real constraints
because in the latter case,
you need to extract these real constraints
from the real world in real time.
And we'll talk a little bit more
about this in a moment.
So, I'd like to show you here
a video of our system working.
And this is actually a Unity screen capture
that shows a scene captured live
by a mounted Kinect camera,
and its overlay with augmented reality visualizations.
And in a couple slides, I'll show you
our actual augmented reality view.
But suffice it to say, notice how the user
is building up virtual roads that are snapping
to be parallel or perpendicular
to the edges of the table.
And, our system extracts linear edges
and planar surfaces in real time for snapping to.
And our results in this paper show
that our system can extract these linear edges
and planar surfaces that people would want to use
for snapping and that snapping in augmented reality
is faster than no snapping for positioning
and orienting tasks.
And we'll also see that enables new forms
of creating virtual content.
So with that, let's talk about the prototype.
So our prototype system hardware is composed
of a Lumus optical CD or head-mounted display,
wireless remote for interaction,
a Kinect camera mounted above the user,
and an OptiTrack system for tracking the user.
But we imagine that this would be implemented
on a fully-contained wearable device
such as HoloLens in the future.
This is the view on the left here
through the optical-see-through display.
And note, due to the HDR nature of the capture system
that we used to capture through this display,
we're mainly going to be showing images like that
on the right side for the rest of this talk.
And again, those images on the right side
are a live capturing of the world from a Kinect camera
that is overlaid with virtual content.
So again, we extract linear edges and planar surfaces
as our physical constraints.
You can think of other constraints
such as curved surfaces, but linear edges
and planar surfaces are a subset that is simple
but it's rich enough and contains a lot
of semantic information.
So the linear edges we extract from the Kinect camera
using the Hough transform followed by some processing.
And then we extract the 3D line segments followed
by temporal filtering to reduce noise
and also adjust to the dynamically changing world
as you can see here in this video.
And this runs in 15 frames per second.
The planar surfaces, again, we run another Hough transform
on the depth image, and we follow this
by greedy agglomerative clustering.
And this runs in four frames per second
in our unoptimized implementation.
So with that, we can now talk about our snapping algorithm,
which is, I'll say it's a vanilla standard
snapping algorithm.
But in this scenario, it's unique.
Say we have our virtual cube here.
And now we have, in this cube I'm calling these
virtual constraints, namely these edges
and these planar surfaces.
So these are the virtual constraints
internal to this virtual object.
Now we want to snap this virtual content to the real world.
So say, and again, this real world has real constraints.
So say we're focusing on the table,
and the table has a set of edge constraints
and this planar constraint.
And now we want to snap this virtual content
to be in this target snapping position.
So this is our goal.
So, the way it works is we snap using the pair
of compatible constraints with the lowest cost.
And what do I mean by this?
So, I'll explain it in the context of the algorithm.
It's pretty simple.
So, we first search through all the virtual constraints
and we check to see if it's compatible
with any of the real constraints.
So, say we're looking at this red virtual constraint.
And we check all the real constraints,
these real ones, and say now we're focusing
on these two red edges.
So, we check if they're compatible
by making sure that they're close enough in distance.
And if so, we use, as the cost, the linear combination
of this distance in that arrow there
and also the distance between these two constraints
to the exact centroid of the real constraint.
And we can do something similar with planar surfaces
such as these two planes.
And then, that's the real plane extending out.
And then we use these two distances for the cost.
And I'll refer you to the paper
for more details on this algorithm.
So with that, now we can talk about some user evaluations.
So, to evaluate the constraint extraction,
there's many different types of lines
and planes you can extract from the world,
but we wanted to see what users thought
would be good for snapping virtual content too.
So we explained a simple scenario such as snapping
your virtual application windows
to different things in this scene.
And we gave users, 40 participants, 11 scenes
and asked them to draw on these 11 scenes
the edges and the planar surfaces
that they would use for snapping in this context.
So, this is an example where they snapped.
This is an example of lines they drew,
edges they drew.
These are planes that one user drew.
And we aggregated this over the 11 scenes.
And our algorithm could detect the majority
of the edges and planar surfaces that the participants
indicated they'd use for snapping.
So then, we evaluated our SnapToReality techniques.
So, the first application was AR content placement
and the second one was AR content creation.
For AR content placement, we wanted to see
how fast users can place, orient, and resize
different virtual content with respect to the real world
with snapping and without snapping.
So, in this example here, you see that the user
is picking up a virtual application window
and then it has a target ghost window
that they have to match first in positioning orientation
as the first part of the trial
and then next in size to resize.
And we gave users 20 trials, 10 with snapping
and 10 without snapping.
Here you see the no-snapping condition.
And notice in this one, the virtual content
does not snap in position or orientation to the real world.
It does, however, it is placed at a distance
that is the depth of the user's gaze intersection
with the environment.
But other than that, we did not change anything
between these two conditions of the snapping
and no snapping.
And we measured task completion time across all these.
We also gave users five training trials
before each of these conditions
and balanced their conditions completely.
This is the view through the optical-see-through display.
And notice, it's actually not that bright to the user.
It's just the limitation to how we captured it
through the optical-see-through display.
This is our second demo operation, AR content creation.
So the user draws out a rectangle
on planar surfaces and can then extrude cubes
to different heights, snapping to different areas
of the environments.
Here you see the user drawing out some virtual cubes.
Now we notice that with this kind of dynamic extraction
of constraints, users can now use different objects
in their environment such as this physical measuring tape.
So first, you see here on the right
that the virtual cube is snapping to the height
of the box and then it snaps to the height
of the measuring tape.
So timing results for the demo application one,
we saw that there is a statistically significant difference
in the time for positioning and orienting.
Snapping took about four-and-a-half seconds,
and then no snapping took it about 11 seconds.
However, in scaling, we didn't see
any statistically significant difference.
And we think this might be due to the scenario
that we chose.
And also, our scaling is actually, if you think about it,
it's a 2D scaling.
It's not really a full 3D scaling.
Something else that is interesting
is that we observed the user's physical movement
between the two conditions.
So, excuse me, using snapping on the left,
we see the user moved a lot less
than without snapping, excuse me, on the right.
And we think this is, this strongly exemplifies
the importance of bringing different techniques
into augmented reality such as snapping.
And we think this is good to explore in the future.
So, there are some questionnaire results.
I don't have too much time to go into it here,
but you can look at the paper for the details.
But, suffice it to say that there's a statistically
significant difference between the snapping
and the no snapping.
So for example, users rated that they could place
objects more easily with snapping
than without snapping.
You can see the paper for more details on this.
Finally, I'd like to share with you
some design considerations for SnapToReality systems
as future developers.
So, first is the limited user field of view.
We don't necessarily mean the limited display field
of view as we saw in the previous talk,
but even with a perfect display,
how would the user know about the constraints
behind them, outside the field of view.
And how would they select the constraints
that are outside the field of view?
One way, here, we offered to suggest
is perhaps the user can actively select
the constraints rather than passively
just moving the object around close enough
to that constraint.
Second is changing the view in AR.
In augmented reality, the users are limited
by the physics of the real world.
If you want to look at something closer,
typically you move up to it to be closer.
And this is something that we should consider
more for the future.
The camera field of view, mainly what I mean here
is the camera has to first sense the environment
to extract these constraints.
So the user must move the camera
or let the camera sense this world
before he or she can use these constraints.
Noise and complexity in the constraint extraction,
the sensor might have some noise.
This environment is very complex.
So one potential solution would be
to have some user in the loop, disambiguation.
So future SnapToReality systems
need to design how they extract these constraints
with this in consideration.
Again, dynamic constrants, I mentioned this earlier.
It might not be obvious at first,
but now that we can extract these dynamic constraints,
we can use physical measuring tape
to create new constraints on the fly,
and this will guide the formation
and the creation of virtual objects
with respect to the real world in augmented reality.
And finally, constraint visualizations,
so, on the left here, you see our visualization
for the edges that you're snapping to.
On the right is for the planes.
For the planes, I'll explain it.
We have a fuzzy circular region
placed at the centroid of the plane
and then a red line connecting the virtual content
to that centroid of the plane.
And one consideration here is,
"Do you visualize all the constraints?
"Do you visualize none of the constraints?
"Or do you visualize just the constraints
"that you are snapping to?"
In this work, we just visualized the constraints
that were snapped into,
but it's something to consider for the future.
And, final thoughts.
So, for future work, we think it would be interesting
to explore interaction via other modalities
such as using hand gestures or eye gaze
for actively selecting the constraints.
We can think about touching the constraints,
speech recognition.
If your camera has sensed the environment,
wouldn't it be nice if you can just say,
"Place this object parallel to the right of the door."
Also, filtering constraints.
If you're snapping to a constraint that's behind you,
you might not want that.
So we need to consider what are other ways
to deselect or undo different types of constraints.
And again, there's more types of constraints out there.
There's curved surfaces, semantic snapping.
One example of semantic snapping is,
say you have a recipes app like in the first example.
Your recipes app might want to be favored
to be placed on the kitchen cabinet
rather than on your fireplace, for example.
So, thinking about ways to bring semantics,
instead of just simple geometric snapping
into augmented reality.
So with that, I'll conclude.
So we presented our SnapToReality concept and prototype,
and we showed that we can extract the majority
of constraints the users say they would use for snapping.
We've confirmed that snapping in augmented reality
is faster than no snapping for positioning
and orienting tasks and enables novel ways
to interact with creating virtual content.
Thank you.
(audience clapping)