- I'm going to talk about our project
on panning and zooming panoramas
in virtual reality devices.
VR devices have created growing interest
in a new generation of user interaction
and user experiences.
Such immersive digital content experiences
enable users to explore things by panning all around.
It is amazing that we can virtually be there,
and get to gaze at the starry vista,
but can we zoom in and get closer to the stars
to observe them better in details in VR devices?
First, this becomes more and more accessible
to create high resolution panoramic imagery
with the development of robotic image
acquisition technology.
In fact, there are thousands of such panoramas
being shared online,
which makes anyone with a browser
being able to zoom in and explore big events,
urban-scape, or the nature intimately.
Each of the imagery contains plentiful details
which are worth exploring.
For example, this is a panorama
taken from the London 2012 Olympics opening ceremony.
With zooming in, user can not only observe the teams
from various countries in a parade,
user can even look at
each single person as well.
High resolution of panoramic imagery can also be surreal.
Artistic are creating very large format panoramas,
combining high resolution imagery
with AI-based deep dream,
to create new artistic world.
By zooming in, this dream-like appearance
is better visualized,
and can be better explored by viewer.
But how can we view them in VR?
As the current immersive panorama viewers
in VR devices only allow users to pan
and decide which direction to look at,
our goal is to enable users to zoom in
and determine what details they want to explore.
Actually pan and zoom is a fundamental operation
for image viewer applications on flat devices.
The scene follows the movement of mouse control,
especially when the mouse drags
from one building to another,
it's actually scanning a very small area
in the huge panorama.
To really more clearly,
we show them in this hemispherical projection here.
When user drags the mouse on the browser window,
the view looks from top right to right right.
We can see that at higher zoom level,
it rotates in a smaller scale.
In other words, the higher the zoom level is,
the slower the panning speed becomes.
So we call it a slower mode.
This is very natural, and intuitive on flat devices.
Then presents the question,
why not just use slower modes in VR?
In VR devices,
the direction of display
is controlled by user's head rotation.
If we also compensate panning to explore
in higher zooming level,
it may result in two problems,
physical challenges
and a proprioceptive problem.
Look at this example.
The user first looks at the Chrysler Building,
and then zooms in
and observes the details
after (mumbles) orientation
by turning down her face.
The first problem is,
when she still wants to pan down,
she physically cannot tilt her head further.
Next, when she zooms out,
now that she's still facing down at her feet,
but she feels like looking forward,
because she's seeing the horizon.
And when she faces up to look forward,
through the screen, however,
she feels like she's up in the air,
because, which totally upset her sense of stability
in this environment.
This is the proprioceptive problem.
Having experienced these problems with slower mode,
we identified the following criteria
for our design interface.
First, user should have the freedom
to look all around the environment,
and zoom in the details in any direction.
Second, for the viewpoint,
the interface should face visuals
with user's real world self motion.
As mentioned before,
as this converting to your (mumbles)
or move the horizon line.
We first designed a straightforward mode,
keeping these properties, which we called normal mode.
In this mode, user controls the direction of view
via the head motion directly.
Independent with the zoom level.
It allows user panning all around
at all zoom levels,
and doesn't move the horizon
after operations of zooming in and out.
Let's see how it works.
This is a panorama we took
on the roof of Facebook office in Seattle.
So here the user first zooms in
and wants to look closer to the people
along the South Lake Union.
And then, even as he pans slowly to the right,
the scenes change too rapid to observe,
and makes the user experience nausea and fatigue.
And when the user is asked
to search and look at the Space Needle,
he got lost, and spent quite a while
to find it in this very zoomed in imagery.
Therefore, navigation is one of the problems
in this normal mode.
Another problem in that,
as the zoom level becomes higher,
the perceptive visual motion becomes faster,
which in this case is eight times faster
than the speed in traditional slower mode.
The mismatch between the slower,
slow head motion and fast display motion,
leads to users' motion sickness.
Third, since the visual scene
changes very fast with the head motion,
user needs to move their head very carefully
if they want to observe a specific spot.
So in normal mode,
it is hard to control scene user wants to see.
Based on those downsides,
we then identified three more properties
for our desired pan and zoom viewer in VR.
The display should match well
with user's perceptual movement
to reduce motion sickness.
Additionally, user should be able to control
where they want to, and expected to see.
And finally, the interface should help user
to navigate in the scene.
With this objective in mind,
we proposed first to limit the field of view
to the center of the screen
in order to displace user's perceptual motion,
and simulation sickness.
Then to place a zoom in center area
in front of a non zoom in background
that provides context in our peripheral vision.
Here we show an example of circle mode.
Though rapid panning still brings strong motion sickness,
when user pans slowly,
limiting the zoom in visual field,
reduces the visual motion,
and the background layer
serves as a stable reference
to alleviate the motion sickness.
The next mode, we modify the circle transparency
based on the panning velocity.
It will allow the background to be visible
through the zoom in imagery,
during rapid head motion,
which reduced the motion sickness a lot.
We also add a crosshair
when the circle becomes transparent,
to improve targeting and the navigating experience.
We also proposed a zoom circle mode,
which adapt a zoom level to the user's head motion.
It zooms out during rapid motion,
and zooms back in as the view comes rest
on its subject.
We conduct a user study,
and asked 33 participants to complete several tasks.
In each task, they were asked to find
a specific object in a scene,
which is hard to notice without zooming in.
Before starting the tasks,
we gave users a short instruction
for the modes we discussed before,
and in each task,
we randomly assigned them one of the four mode,
and don't allow them to change it.
And we recorded time spent by participant
to find this target.
Here we show the histogram of completion time
used in normal mode.
Considering several participants
had no VR experience before,
and can suffer from different level of motion sickness,
we allowed them to quit a task
whenever they feel discomfort.
Here for visualization purpose,
we show the time as a 180 seconds
if they quit a task.
In normal mode,
there are 16 out of 89 subjects,
22 give up.
And the median of all the completion time is 95 seconds.
Here are the histograms for circle mode.
Fewer subjects gave up,
but there are still six out of 93,
and the median time is much lower
than that in normal mode.
In both alpha mode and zoom circle mode,
there's no give up,
and the median time is even slower.
In this figure, we plot a median distance
from user's pan window to target
along the time in each mode.
We can see that the distance of alpha circle
and zoom circle mode drop significantly faster
than the other mode
during the first 20 seconds,
which suggest that
most of them helps the navigation.
Here the solid line only includes subjects
completing the task,
whereas the dash lines
includes the give up data,
and meet the task as well.
Note that the green solid line
in the normal mode start at the distance
much slower than others.
This suggests that the user that starts far from the target
are more inclined to give up.
We also collect several qualitative data
by a questionnaire.
The score arranged on a Likert scale
from one meaning bad to five meaning excellent.
Right now the aspects we are concerned about
is whether subjects experience motion sickness
as using this interfaces.
From the histogram and the mean scores showing here,
we know that narrowing the field of view
slightly improves the movement comfort,
while adapting transparency or zoom level
with panning speed
significantly helps us reduce motion sickness.
We also asked how easy it is
to navigate to other area,
and target to objects there.
And the score shows a similar pattern,
and alpha zoom circle, and zoom circle mode,
still outperform the others.
Finally, we asked which one they prefer to use
to explore new panoramas.
The result also looks similar.
Based on both the qualitative
and quantitative result,
we come to the conclusion that normal mode
is not suitable for pan and zoom imagery,
wide circle mode is slightly better,
alpha and circle mode are assessed
significantly better than the other two.
People like this mode in different scenarios
for different reasons.
Some people prefer alpha circle mode
when they know where to explore next,
but when user is not familiar with the panorama scene,
one may prefer zoom circle mode
to gradually find interesting object
or area to explore.
In summary, as VR devices become more popular,
it is interesting to rethink
how we can solve the traditional tests
which are well-studied on flat platform.
Panning and zooming high resolution panorama
discussed in this paper is just one example,
and this leads us to some ideas for future work.
In this paper, we haven't delved into
stabilization technique,
or gesture interactions in the interface,
and both of them could improve the user control
and viewing experience, furthermore,
it is also an interesting problem
to extend the pan and zoom interface
for stereo panoramas.
This comes to the end of the talk,
and thanks for listening.
(clapping)
- [Questioner] Hi, Consuelo Ramos,
I'm a professor of research.
I wonder as virtual reality becomes
more prevalent, and you actually have synthetic scenes,
not just a panoramic imagery,
that zoom is not much a zoom,
it's more as I'm actually getting closer
to the area I want to see, and then,
looking around, is just looking around.
Do you see the same techniques applying,
or just not necessary,
potentially for those things?
Or you may be able to synthesize
what a panorama may look like
if you were closer, actually?
- So your question is why is this different
in 3D videos, than in simulated scene, or?
- [Questioner] I know why a panorama is different
than actually 3D scene, because very complex
and dense 3D scene,
but I'm wondering if you think the lessons
you learned here
carry forward,
or perhaps there's this interesting idea
of can you synthesize a panorama
from the point of view of being closer
to what you want to look at?
- Oh yeah, actually we also try
imagine this panorama as a 3D sphere,
and we just like move the object,
like originally in center,
and move it closer to the sphere, to the edge of sphere,
and it will get distorted,
because if you look forward,
it is actually get closer to the panorama,
that if you'll look around,
it will, the closer it is,
the scene is stretched much larger,
and the further, it is stretched
compressed, like suppressed, like littler,
and you don't know what to do at that point.
So that's a problem.
We also try other possible ways to deal with this,
this panning and zooming,
either with,
but nothing works great better than this circle mode,
so we stuck with this one.
Thank you. (laughs)
- [Another Questioner] Hi, I'm (mumbles)
from (mumbles) South Korea.
Very interesting talk,
and a very practical one too.
This might be a little silly question.
Is there a reason why it has to be a circle,
and not a square, or any other polygonal shape?
Thank you.
- Nice question.
Yeah, during this study we also tried,
like first to answer your question,
why circle is more intuitive,
because we imagined this problem,
as like you put a telescope,
like looking on the top of a building,
it's very similar to that case,
so it's more intuitive to users,
and we also try other, like,
transition between the edge,
like more transition, that is almost the same performance.
Yeah, thank you.
- [Questioner] Hi, Tim Dwyer, Monash University.
I was wondering if you tried to stabilize the image at all
when you're in zoomed mode,
or whether that would even be possible.
- Oh thank you.
So actually I haven't mentioned in the talk,
but we, before trying circle mode,
we already add them being, effect to,
help with the stabilization.
That yeah, it still doesn't, like good enough.
- [Questioner] So I see, so this is already stabilized.
- It is already stabilized,
and is talked like in details in the paper, thank you.
- [Another Questioner] How did you decide
the size of the circle?
And why there's four modes?
I mean, like is there any meaning for each mode?
- Yeah, thank you for the question.
We tried like a pilot study,
to ranging the size of circle,
and it's a balance between the field of view
and the motion sickness with that,
so we finally choose this one, thank you.