- Thank you for the introduction.
This is the work I did at NYU,
where my supervisor Daniel Wigdor and I
were just in New York, we were wondering
whether we can do something cool.
We just hang out in in Ken's lab
for six months and we did this.
So, hope it's cool to you.
In recent years, we see a lot of applications
for virtual reality, including gaming, education,
story telling and many others.
And the immersive 3D environment,
coupled with properly designed content,
really allows the user to directly interact
with the experience, with physics and mechanics,
are only limited by our creativity.
Well, the immersive experience itself can be direct,
but occasionally it can be a little bit intense,
as we saw in the Ready Player One.
Here, this little girl is really frustrated
because she lost a bunch of things in the virtual world.
Well, all these things is direct and frustrating in the art,
it is equally indirect and equally frustrating
when you're designing such content,
or designing 3D content in general.
In addition to having to use all these powerful,
but really complex softwares.
The developers, while freezing all these flat 2D screens,
they have to imagine what the Colosseum
will feel like in virtual reality.
But the problem is, what you see in 2D,
is not what you feel in 3D.
And besides, how come they get
to hang out with their friends,
but we just have to sit there alone,
stuck in our chairs and work alone?
Now, our lasting impact of words,
winner sits there and she said,
"We need to do work together."
In recent years, there is effort of enabling individual
and collaborative content creation in virtual reality.
However, many individual functionalities simply reuse
the desktop UI, which doesn't work in VR,
and mostly folks are Colosseum viewing
for multi-user collaboration.
The goal of Spacetime is to enable the user
to directly create 3D scenes in virtual realty,
individually and collaboratively.
So we folks are providing interaction techniques
that addresses many challenges
of working in virtual reality.
And we seek to provide fluid content creation experience
with three interaction concepts:
container, parallel object and avatar object.
Let's start with container.
One of things that you will realize immediately
when you designing a scene in VR is,
you need multi-selection.
Imagine I'm trying to move this table,
and all the things on it, against the wall.
Well, without multi-selection,
now I have to gradually grab them one-by-one.
This is really tedious and frustrating,
but also disrupts the layout of the original objects.
With Spacetime, you can create a container
so you can select a set of objects,
either by point and click,
or brush through the objects that you want to select.
And then, you can use the container as a proxy
to move all the objects together.
But, selection technique wouldn't be interesting, right?
You can easily find another 100 lit. paper on this topic.
What we want, folks, is beyond the selection,
but on the context of the selection.
Our insight is that, the interactions relies on the content,
but also create a context.
And the user should be able to interact based on the context
and revisit the previous context for iterations.
Well, reusing content,
or revisit the content is relatively easy
on a small 2D display,
because all the content is within your arms reach.
But in virtual reality, you can just be a tiny speck
of this scene, and the content you want to reuse
are you want revisit my other stuff,
at the other side of the city.
In Spacetime, we seek to extend
the container beyond multi-selection.
When a selection is made with a container,
it not only shows the objects that are selected,
but also preserve the context.
In this case, the spacial context.
And we can see all the objects around it,
and the space around it.
So why is this useful?
Imagine I'm trying, in this room I want to reuse a desk
that I designed in other building.
And, I don't have to go back to the building,
like far away, try to find the desk
that I want to use and move it here.
I can call out all the containers that I've made,
but the thing is, the container preserve the context.
I can see the new content
that has been added to the selection,
while they're really not added to the selection yet,
but I can see they're there.
So here, I notice that some new file folders
have been added to this desk.
And I can easily add this file folder into my selection,
and this forms a new context that I can reuse in this room.
But sometimes, you may just not want to reuse a content,
or you want go back to a content
that you have designed,
and revisit it and iterate on it.
Because the container in Spacetime preserve the context,
we can easily teleport into the container,
and that will take you back to
the position where the container is created.
So here, I see my collaborator here.
Another benefit of virtual reality is,
we don't need to be bounded by
the physics of the real world,
where we are just roughly the same size.
Well here, we have an anima avatar that's super huge,
the other is really so small.
This shows one avatar works with a car,
the other one can be as very big to work on a building.
Also, imagine we are trying to design this room.
I can be very big and put up this screen,
and another avatar can be small,
and fiddling with the chairs.
But the problem occurs when
we're trying to work on the same set of objects.
Here, this user is trying to grab that building,
but he's not able to grab it,
because there's another user have already grabbed it.
But, he's just so tiny and behind the building,
that the other user is not aware of his existence.
I'm not happy with this design.
This is my version of the model design.
And this can be even worse
when they are so stubborn of their own design.
(audience laughter)
So here, they just take turns to grudge,
to move the things back and forth,
and this turn taking is really time consuming
because we have to wait for each other to do their design.
But the problem is, they also block my design, right?
I want to build more things beside the building.
Also, every time I move the building,
I disrupt his design, so this is really disruptive.
When Ivan Sutherland pioneered the virtual reality,
he said, "The ultimate display would, of course,
"be a room within which the computer can control
"the existent matter.
"Handcuffs will be confining,
"and the bullet will be fatal."
If this is the vision
that the haptics guy are trying to work towards,
if a virtual bullet can be fital,
we really need to solve this problem for this two users.
(audience laughter)
What is this problem?
We think this problem is that,
we just think that an object can only exist
at one state at one moment.
This is how the physical world works,
and this is how many of the collaborative system work,
where one user is working on a object,
it's just locked, the other user can't do anything about it.
But we don't necessarily need to follow
this rule in virtual reality.
Why don't we let them just get what they want immediately,
so they don't have to wait for each other?
In Spacetime, they can parallelly grab parallel versions
of an object and just go ahead and place where they want.
We call it parallel objects, so of course,
This video clip is a little bit staged, right?
In the realist it lists your workflow,
you want to say, hey, come here,
let's grab a parallel version together.
In the realistic setting,
the things that you want to interact with,
might already be moved by another user,
and you are just trying to select and grabbing,
grab a moving target, and that would be difficult.
Here, I'm trying to select these buildings,
but the other user have begin to select it and move it.
But the Spacetime can detect my intention of
working with these objects
and spawn parallel objects as a space,
so that I can select them.
I can go ahead and select the objects that I want,
but the problem is,
I want to select them with the context,
where it was with the trees.
So, with Spacetime, as I said,
the container preserve the context.
They are only preserve the spacial context,
but the temporal context, so here,
I can spin the container to roll them back,
so that they are with the trees,
and I can select them with the trees.
Now I get what I want, but these two buildings,
I haven't select them,
which means I haven't indicated any conflict system,
so we just gradually disappear,
because there is no conflicts.
The other user, when parallel objects are created,
he will receive a notification on the container
that he is using to indicate that there is a conflict.
And then, this two user can start to talk,
engage a conversation and start to figure out
which version they want keep.
So, let's back up a bit.
Without parallel object, user A put up their own design,
the other user might not be happy,
and because they are disruptive,
so they have to go back and forth
to figure out which one works best,
and then mutually resolve the conflict.
With parallel objects,
each of then can simultaneously achieve
what they want,
and then they can just dive-in to
the topic resolvement right away.
But, conflicts, essentially are just
different design choices, right?
And design choices are good.
You have multiple options.
You can choose the one that should work best.
In Spacetime, we can explicitly create different versions
to play with the different designs.
So here,
ah,
my video is now playing.
So the space-time continuum for this video just stopped.
(audience laughter)
I can select these buildings
and I can create parallel versions
just to experiment with the different designs
that I want to play with.
But the other user could just start work
on one of the parallel versions that I left for him,
so to configure the other design.
With all these three designs,
I can easily navigate them
by just toggling the parallel containers in my hand,
and trying to see which one work best in the scene.
But when we're comparing different design choices,
there are multiple ways we want to compare them.
Sometimes you want to compare them
just by putting them, itself,
in the scene to compare it with the environment.
Sometimes, we want to show all of them at full transparency
to see which one, to compare within themselves.
Spacetime support both of the comparison
to allow the user pick the one they want.
So this is a parallel object.
It's our solution to enable users
to explore multiple design variations,
and to handle conflict when there are multiple users.
But in a shared virtual environment,
conflicts not only occur of the design
that you are working on,
it also occur between people.
Most of the virtual reality environment
support teleportation.
Allows the user to quickly navigate a large space.
This is usually fine.
Here, my colleague and I are discussing about this park.
And he realized there is another park that is interesting,
but it's on the other side of the city.
He just teleport.
And, to me, he just gone!
Now imagine you're having a conversation
with the person sitting to your right,
and you say, I'm a researcher from what are you institute,
and he just disappear.
That would be really impolite.
(audience laughter) And that's super bad.
You will be angry, right?
So what is the problem here?
I think the problem is, well, in the physical world,
we are bounded by the physics,
where we know where the person will be in the next moment.
But when we giving teleportation to the users,
they can just be anywhere, in any moment.
For me, when I'm looking at him,
I have an expectation of where he will be,
and he, in his mind, he has another expectation
of where he wants to be next.
So this is actually another design conflict.
It's a conflict of the user's position.
When we have conflict, what do we use?
We use parallel objects.
Here, whenever user teleports,
he will leave a parallel object,
well, parallel avatar of himself there,
which I can use to select to teleport
to where he teleports to.
And I can just follow him to this park
on the other side of the city.
What also is implied here, is that each of the avatar
is no different from any other objects in the scene.
You can leave parallel avatars when there is conflict,
but you can also explicitly create a parallel avatars.
For example, I have just designed this amazing building
and I really want to show it to my collaborator.
I can go find him, well, this is, and technically,
I didn't introducing him, check on paper.
(audience laughter)
Here, I want to grab his attention.
I can't really grab him as I grab other objects.
Just thinking that another person sitting beside you
are just grabbing you and move you around.
Causes serious motion sickness,
because you're just losing the agency of your body.
You know, what's the problem here?
The problem is, I'm trying to get his attention.
I have a expectation of where he needs to be.
He needs to be at the building to see my design,
with that big Visa logo.
But he, because he has the agency of his body,
he has expectation of his position.
What I can do, is I can just grab the parallel
version out of it, because he has a version of it,
so I grab a parallel version.
And I can position and scale this parallel version
to define a view for the other user.
Of course, this is a conflict,
and he will receive a notification, say hey,
here is a conflict, this is the real destination suggested,
by another user, and he can just teleport
whenever he wants to, to go to the view
that I have defined for him.
You don't always need to go find him and grab him.
You can use containers.
You can use containers to select him,
and take him to the other world that you're working on.
And put him out in the world so that he can just come
to the world to work with you.
Okay, so I have introduced to you container,
parallel object, and avatar object.
I want to emphasize that these are not irrelevant concepts.
They are connected.
That rich inter-connection.
These concepts really allows us
to address a serious problems in VR.
We can have parallel containers.
We can have parallel avatars.
We can have parallel avatars in parallel containers.
This just opens up a lot of space for exploration
of new ways of creating content in virtual reality.
So with this, I would thank you for your attention,
and I'm happy to take questions.
(audience applauding)
- [Moderator] Again, microphone's at the front for questions
- [Male Audience Member] Hi, I'm Jacky,
I'm from Center University.
Great work, great talk.
I wonder when you design a system,
do you start from gathering problem off existing designs,
or you have some kind of a way of thinking this problem,
and then find out there's a few specific problem
that you try to solve in the system.
- We didn't really grab a bunch of people
into our lab and try it out,
because there isn't really full featured
collaboration system out there.
So what we did is, we do this iterative design
where we build a version and then we grab just people,
like there are a lot of visitors came to lab.
Just let all of them try them out,
and then trying to get their feedback
and gradually working towards the version that we want,
to address their problem.
- [Male Audience Member] Thanks.
- [Female Audience Member] Hello, I'm Kendra,
from University of Calgary.
When I'm thinking about parallel objects
and kind of conflict management, I think about Git,
and kind of how complicated that can get,
because sometimes my changes are small
and go well with your changes.
We can just move them around
and sometimes they're in direct conflict.
Let's say you create a parallel object.
Are the changes that I'm making in the one,
my original version, also being copied into yours,
or is that now just sync date and time
from when I copied it?
- It will stay where you are.
It's sill with that version.
This is just our current prototype.
You can easily say, I want to barrow cost to my design
to other parallel versions.
That will require agreement
from other user to accept your design.
Yes, it's very similar to Git
and then the good this is
that we can leverage all
the people have designed for Git.
Just borrow the concept into this.
What we tried is it's really realtime Gitting,
sort of like coupling management
it's really from the key here.
- [Woman Audience Member] Okay, thank you.
- Thank you.
- [Moderator] Alright, let's thank Haijun.