So I'm Jackie [INAUDIBLE] from Stanford University.
Today, I'm here on behalf of my co-author,
Christian [INAUDIBLE],, [INAUDIBLE],,
and Andy Wilson to present, "DreamWalker--
Substituting Routine Walks in Real Life
with a Virtual Reality."
So you have been here for a long time.
A lot of talks about VR.
VR has bright futures.
So we envision in the future, people
will spend way more time in virtual reality, right?
Like in trains, on the ride, there's
me on an airplane playing VR, and on buses.
But in all of those situations, we remain static.
So we wondered, how can we be able to enable user
to play VR the rest of the day, even if they're walking,
they're actively on the move?
So that's why we propose our solution, DreamWalker.
So DreamWalker allows users to enjoy a walking
experience in VR, while completing real-world world
walking task.
For example, a user is going to his morning
walk to a different building on campus.
And this is what a user sees on the VR headset.
Here's New York and [INAUDIBLE] in a big city,
while walking a pretty tranquil street,
and the user doesn't really see this stuff
in the real-world world, just walking in the virtual reality.
So this is a [INAUDIBLE] on the left.
Like, we're generating virtual humanoid
to keep the user safe in the virtual reality.
So let's go into details about what's
happening behind the scene.
So "DreamWalker" is a huge system
with lots of little components.
So we want to give a quick walkthrough about how
this system feels like.
So the users start their dream walk
by starting a destination in the real-world,
and our system automatically plots a path in the real-world.
Then it routes another--
it actually browses through a library of virtual scenes
for a similar path that is close to the path in the real-world,
and is [INAUDIBLE] redirection constraints.
And then the user just walks.
You can see virtual people walking around the user,
and they kind of keep the user safe.
So we shift the [INAUDIBLE] by using our unique tracking
pipeline with inside out tracking has that.
There is also depth cameras, and there's also dual-band GPS,
and finally, really powerful GPO backup computer
to process all this data, and then transmit the VR
image to the user.
So let's talk about Related Walks.
There is Oasis, which is by [INAUDIBLE] and our colleagues,
which generates a virtual environment for statically
scanned real-world space.
There is also [INAUDIBLE],, also from Microsoft Research,
which allows users to walk in in an indoor space
while playing a [INAUDIBLE] generated game.
They have a compelling VR experience,
and they are available in controlled indoor spaces.
They use [INAUDIBLE] generated scenes
to feature real-world world geometry,
however they usually require pre-scanning environment,
and they support mostly only static spaces.
As comparison, DreamWalker is not limited to indoor spaces,
and walks with both unseen and unscanned environments.
This means that DreamWalker handles
a variety of different and controlled objects,
such as pedestrians, cars, bikes parked on the street,
and dogs--
maybe not dogs running around, but definitely
a lot of other people walking around.
It uses preauthored VR environments,
which is quite different from prior work.
And even if the geometry is different
from the virtual reality, we use redirection to make
sure the user always walks both on the path in the real world
and in the virtual world.
So this is what the system looks like.
It has many components, as you can see.
DreamWalker refers to path planning.
It also does environment sensing about all the obstacle
and also locates the user using different sensors that we have.
We also have a static optical representation representing
all the static obstacles.
From the environment sensing, we got optical,
and then we represent those adhoc, and then any obstacles.
We fuse all kinds of sensing data
we get [INAUDIBLE] sensing, and then position
the user in the real world.
And then we render a VR scene to position a user in there,
and then we do [INAUDIBLE] walking
to make sure the user is walking safely
in both real-world path and virtual reality path.
So let's start by talking about path planning.
As you've seen before, we already
have the path in the real world, which
is going to find the correct path in the virtual reality.
The way we do that is we first approximate the path
in a real world and virtual world as line segments.
So even if there's curvature, we just approximate them.
And you must find a matching path
according to a few coherent criterias,
including they have to have similar length,
they have similar shape or curvature,
and they have similar angle between segments.
And then we're going to talk about obstacle detection.
Before I go into details, I want to emphasize
a few different obstacles that were identified in DreamWalker.
So there's static obstacle, which
is just like buildings and stuff, and also
huge plantation and stuff.
So they're actually registering the map.
So usually when we're doing path planning,
they should be already registered.
So if the user walks on the path,
they should never walk into them.
There's also adhoc [INAUDIBLE],, which
is like smaller obstacles where it
won't be registered by the map, and then they're really small.
So you have to use our real-time environment sensing system
to detect them and make sure the user walks around them.
And finally, there is dynamic obstacle, which is basically
humans walking around.
So a pedestrian moves pretty fast,
so we cannot detect them using basic height-based method.
We have to use a deep learning-based method
to detect them and project them onto the 3D space
using the point cloud.
For representing a static obstacle, as I said before,
we just make sure the user walks on the right path.
The way we do that is we pre-populate the environment
with all kinds of obstacles, and then
once the matching path is found, we just
remove the obstacle on that path so that the virtual environment
is left with only one path open and all the other paths closed,
so that the user doesn't walk into the wrong path.
So environment sensing is a really important part
of DreamWalker.
It incorporates all kinds of different sensors.
It has dual-band GPS, inside out tracking, and also dual-RBGD
cameras.
And the way we do that is we start with the RGBD images.
So we have two entire [INAUDIBLE] 435 cameras
with a combined field of view of 86 by 98 degrees.
And it can sense stuff up to 10 meters.
We first fused the RBGD image and adapt image
as a point cloud projection with the 3D world.
Then we projected the point cloud onto a plane
where we built a height map.
So in this height map, you can see almost all
of that obstacle is highlighted in this kind of bright
[INAUDIBLE],, which represents really high gradient.
And this is a really useful key for us
to register which objects are obstacles, which ones are not.
And also there's color map.
Color map will project a color point cloud onto a 2D plane,
and then we accumulate them over time to project a color map.
As you can see, in the color map,
you can clearly see the edges of the pavement, which is also
very important because sometimes,
when we walk off the pavement, there's
little height difference, but a huge color difference.
So we also use that as a clue for detecting obstacles.
Finally, we constructed a ground map.
The ground basically is what we call, walkable areas.
So we want to know which areas are walkable,
so that we only get the user to that kind of area.
The way we construct that is we assume where the user is
standing is definitely ground, and we [INAUDIBLE]
iterate a flat view from that point.
And then if there's a rapid height change,
it's probably an obstacle.
And If there is a small height change,
but there's a huge change in color,
then it's probably off the pavement.
So we also recognize that as an obstacle.
So this is kind of expensive because iterating flat view
takes some time write.
So we only write a few iterations every frame.
Finally, we construct an optical map
as the edge off known ground.
So as you can see, we only put obstacle
where we know the ground is, whether it is ground or not.
If it's really far away, we don't get accurately enough
data there, we don't put obstacle
there so that the user doesn't see other obstacles just
in front of them, so that they get
confused of what their path is.
To make user do this safely, we have
to do everything in real time, so everything's GPO
accelerated.
One exception, as I said before, is human,
which they move really fast, we cannot construct a very
reliable height map.
So we use YOLO to tag their position
and map them using the point [INAUDIBLE]..
So we have static [INAUDIBLE] field, ad-hoc field,
and dynamic obstacles.
We will represent them using--
like the static is using the facade and cars
and other static obstacles, and ad-hoc and dynamic obstacles
are represented by humanoids.
So we generate a few humanoids, and they
walk towards the position of the obstacle,
so as long as you don't walk into a humanoid,
you won't walk through any obstacle.
And so we have an obstacle, but we still
need to guide the user to the correct position.
So this part is what we call, the real-world positioning.
Any of given existing sensors is not accurate
enough for our purposes, we need the user
to walk a few minutes walk, while maintaining centimeter
accuracy because they need to not get sick in VR.
The first thing we do is, we have
a GPS which is slowly accurate, but remains consistent
over time.
And we have inside out tracking, which is fast, accurate,
but drifts over time.
We do iterative closest point to map the inside
out tracking data onto the GPS data,
and then we basically project the VR tracking space
into the GPS tracking space, so that you
can make a consistent tracker over time, while you can also
give us fast and accurate response.
This gives us accuracy around 2 to 3 meters,
while we actually need less than one meter
of accuracy, especially during turns.
So what we do is we actually permute around a lot
of different position and orientation
of the user's position, and then we
find that the one position where the path can feed out
of the ground mapping.
The assumption is, the user's path is going
to be on the ground, right?
So then we just find the one path that kind of
feeds the ground map.
And this thing's really computationally intensive,
but it's GPO accelerated so that user
can walk safely in real time.
Finally, there's redirection that has to happen,
otherwise the user cannot walk on the real path
and the virtual path at the same time.
So let's say, this is like a real path that the DreamWalker
path [INAUDIBLE],, and this is a virtual path
that we found in our virtual reality libraries.
And for example, when the user's position is right exactly
on the path, that is fine.
We can find that position in the virtual reality, no problem.
The problem is when the user is a little bit off the path.
This doesn't mean that user goes to the wrong path,
but just means the user is not exactly
at a center of the path.
In that case, we got a pretty confused.
So I let the transformation of the next segment
to compute the current location where the previous segment
to a computer current location.
It's really bad if the user walks around,
and then their position just suddenly
jumps just because they're closest transition
from one segment to the other segment.
So we need a mapping so that we can have a transformation that
is continuous across the space.
So we actually use a really old method, incoming to graphics,
which transform y image to another image using
a few defined edges.
So in this method, the transform of any given point
is a weighted average of transform
of all of the segments.
And then the weight is inversely proportional
to the minimum distance of the segment.
So we basically do the same thing for this.
If a user positions here, and then we
bring the computer distance between user position
are off the segments and within compute weighted average of all
of those transform, and transfer user's position
from the real world to virtual world.
And then this transform guarantees
that when the user is walking around your continuous path,
the user's passing virtual reality [INAUDIBLE]
is also continuous.
And next step, so we have the transformation from the user's
position to the GPS location.
We also have the user's position after the redirection.
The problem is, this transformation is not always
stable, especially where the closest point, the user's
position.
Like, this transformation is jumping around, which is bad.
We don't want that to happen.
So the way we do that is we first
have the current position of the user.
You have it from user movement a little bit, right?
So they expect themselves to be in another position
in the next frame.
We'll have the target frame is kind
of what we want him to be so that he walks on the real path.
The way we do that is we, in outer frame,
we try to move the expected position a little bit closer
to the target, so that what the user is actually seeing
is the [INAUDIBLE],, where we make
sure that is within the other redirection threshold that
is investigated in prior work.
So this is basically what a user experiences.
They can just walk in Manhattan, avoid a humanoid,
and they can reach their destination.
So we really want everyone to be really safe.
So we almost ran DreamWalker everyday
during the last few weeks of my internship.
This is a few paths we tested.
And then we tested in two different virtual environments,
Manhattan and Viking Village.
And those were we tried three different guiding techniques,
humanoid, as I introduced before, Pac-Man,
which just shows a few yellow dots in front
of where the user needs to move, and also
traffic cones, which shows traffic cones whenever there
is obstacle.
So for example, this is a few seconds
of me walking in the second path.
You can see the full video online.
So finally, I'm going to briefly go through the evaluation.
We asked a participant to tryout our system.
We presented them one condition, RGBD condition,
where to see the RGBD point cloud after five meters.
And in the other condition, they see DreamWalker.
And they walk for 600 meters or so,
and then it's around 9 to 15 minutes.
This is a result. People report DreamWalker
had higher precedence, they enjoyed DreamWalker more,
and then people actually surprisingly walked faster
in DreamWalker condition, but they rate RGBD as safer,
probably because they can still see everything
in the real world.
Finally, I want to recap.
So we present DreamWalker, a pretty huge system,
that basically have a real-time tracking and positioning
system by fusing three different tracking method,
inside out tracking, dual-band GPS,
and RGBD step sensing images.
And we also have obstacle detection system
that detects surroundings, and static,
and dynamic obstacles from a moving
and very unstable hand mounted RGBD [INAUDIBLE] camera sensor.
This is essentially a self-driving human,
in some sense.
And third, we have to experiment with three obstacle avoidance
and redirection rendering techniques that guides users
through two different pre-authored virtual
environments.
With that, I'm happy to answer any questions.
[APPLAUSE]
Great.
I think we have time for a question.
Hi, I'm [INAUDIBLE] from UW.
It's really interesting work.
And I have a question about the obstacles.
So you mentioned reconstruct the obstacles
before you, in front of you, and near you,
but have you ever considered, like, on the ground?
If the ground shape is not flat, like,
there is a cave in front of you?
Right, that will get picked up by the height map.
As I so, like--
height map?
Sorry.
Not-- here we go.
Yeah, so in a height map, if there is a sudden height
change within this obstacle.
So no matter whether it's going up,
whether it's an obstacle on the ground,
or whether there is a pit hole in the ground,
it will all pick that as an obstacle.
OK, thank you.
No problem.
Hey, quite an exciting work.
So I had a quick question.
So what were the responses of those
who are walking in the regular environment like?
So are they actively holding the VR headset?
And is there any scope for extensions on the VR system
that could better guide how other users can walk?
What was that again?
OK, what does the perception of the external users
towards the VR user when they were using DreamWalker?
Oh, yeah.
Right.
Yeah, so while we were testing, I think most of them,
they were just really surprised that this person can
walk safely, even if they have the VR headset on.
They're asking whether or not people
can see the real geometry.
I tell them, no.
And they're really surprised at how this could ever happen.
A lot of these people like--
whenever we had to do an experiment,
all the people were just watching
us doing these very kind of--
although, a lot of us do walking.
So like we're still testing the detecting other pedestrians.
But I think in the future, if everyone's using this,
maybe there's even better ways to detect obstacles.
For example, you kind of know where the other user is,
so you don't even need to pick it up using your [INAUDIBLE]..
Very good question.
All right.
Well, let's thank Jackie for that great and memorable
presentation.
[APPLAUSE]