- This project was carried out with Hrvoje Benko
last summer at Microsoft Research.
And thank you very much for the introduction.
I'd like to talk today about field of view.
This is a diagram of the binocular field of view
of the human visual system.
This is from Jones's IllumiRoom paper.
You can see in the middle, combining the green,
yellow, and orange areas, that's the foveal area,
the part that senses detail and color.
On the outside, the red and blue areas,
that's the peripheral vision.
That's the part that gives you context, motion,
and notification.
The peripheral vision serves very different functions
from the foveal area and has a lot less spatial resolution.
The total field of area is around 190 degrees horizontal.
When we're out and about in physical reality,
the peripheral vision is constantly stimulated
by the world around us, providing us
with excellent spacial awareness and motion detection.
But by contrast, modern virtual reality systems
with horizontal fields of view of around 90 to 110 degrees,
are like looking into binoculars.
You can see that even 100 degrees horizontal field of view,
denoted by this purple box here,
discards huge swaths of our visual field.
Although users might feel as if they're immersed
in the virtual environment,
they're really getting a kind of tunnel vision.
The areas outside the VR display space are just black.
This reduces spatial awareness and can impact immersiveness.
In current-generation augmented-reality systems,
the situation's even worse.
The field of view is even narrower,
down to 40 degrees horizontal,
just that little tiny purple area in the middle.
And indeed, 40 degrees is actually the upper end
for current commercial AR devices.
Naturally, of course, wider fields of view
have been long sought after in the literature,
but there's three primary obstacles
which have prevented widespread adoption.
First of all, wide fields of view
demand specialized optical setups
to extend the display into the peripheral regions
and wrap the display around the user's head.
Existing approaches to go beyond 100 degrees
include things like tiling displays,
complex free-form optical designs
like the catadioptrical setup used by Nagahara et al.
on the left, carefully designed Fresnel lenses
like those used in the HTC Vive
and this StarVR prototype in the middle,
or novel light field displays
like Maimone's pinlight displays.
Secondly, wide fields of view demand a lot
of computer power to actually render.
A high-resolution full-field-of-view image can be
three times as many pixels as a standard field of view.
If you combine this with the very high refresh rates
that are required for flicker-free visuals,
this results in a significant computational cost.
Finally, human physiology presents a third major barrier.
You've all heard about motion sickness
from the previous talk,
and we talked about the vergence accommodation conflict,
but the other major conflict
is the motion sickness conflict.
Your body gets its sense of self-motion
from two different sources: the visual information,
which is basically optical flow across the periphery,
and the vestibular information, consisting
of physical motion that's sensed by the inner ear.
And in VR those don't always coincide.
When you're moving your head around in VR,
your head's moving, so you get vestibular information,
and the visuals will update at the same time,
so there's no conflict.
But if you move yourself around, like with a joystick,
only the visual periphery will actually see that motion,
and the result is what we call vection.
That's an illusion of self-motion
that has no corresponding vestibular input.
High levels of vection can lead to motion sickness,
and if you extend the periphery,
that only makes that worse.
Our solution to these problems
is the sparse peripheral display.
It's a field-of-view extension
that consists of a sparse array of LED lights
positioned in the visual periphery.
We developed two different prototype devices.
The first prototype, dubbed SparseLightVR,
was based on this Oculus Rift DK2 virtual reality headset,
to which we added 70 LEDs
to form the sparse peripheral display.
On top of that we covered the LEDs
with a translucent Delrin plastic diffuser
to soften and distribute the light.
Here's what that looks like in action.
In this prototype the rendering is all driven
from an external computer,
and tracking is provided by an external tracking camera.
Here's what the field of view of the Oculus VR looks like,
that purple box in the middle,
and here's what that visual field looks like
when augmented with our SparseLightVR LEDs.
Note that the sparse peripheral LEDs cover 170 degrees
horizontal field of view.
Our second prototype, which we call SparseLightAR,
was built on top of a custom AR device
from Microsoft Research, featuring a 62-degree
horizontal field of view.
This AR device is a modified Galaxy Gear,
driven by a standard Galaxy S6 smartphone,
mounted on top as you can see.
And the display is routed through a pair
of curved, collimating mirrors at the bottom,
which reflect the light into a pair of half-silvered mirrors
to create the illusion of images floating in mid-air.
On the left side is our IMU unit
that comprises an accelerometer and gyroscope,
and the tiny little Arduino,
tucked away at the very left side,
which drives the sparse periphery LED array.
To this design we added 112 LEDs
to form the sparse periphery
and covered them with a paper diffuser,
and we used paper here instead of plastic
to reduce weight and bulk.
Here's what that all looks like when combined together.
Here's a simple AR scene.
This prototype is fully self-contained.
The display and LEDs are all driven
off of the smartphone itself,
which renders everything in real time.
You can see that here you have the butterflies
that end up in blue and the bouncing ball
that ends up in red.
Here's the field of view of our AR device,
with the LEDs, to show that we have
about 190 degrees horizontal field of view.
Sparse peripheral displays, as you can see,
are inexpensive, very simple to manufacture,
and can be retrofitted into existing VR and AR displays.
They provide a very wide field of view
without running into the same problems
as traditional approaches.
Because of the low spatial resolution
of our peripheral vision,
they don't require quite so many complex optics
to focus on the sparse periphery.
And rendering these LEDs is also a lot simpler
and can be done at very high frame rates.
And finally, as we'll show in our user study,
sparse peripheral displays can be designed
to reduce motion sickness, even when compared
with conventional, non-wide-field-of-view displays.
Now I'll explain how this all works.
We implemented everything using the Unity game engine,
using the Oculus VR library.
We augmented the default Oculus camera rig
with a set of light probes.
Those correspond to the physical positions
of the LEDs relative to the eyes.
Here you can see all the light probes we used
for the SparseLightVR prototype.
Note that there are two sets of light probes,
one for each eye, and here it looks like
they're actually overlapping, just because
of the rendering and because of the scaling from the camera.
But in practice they're viewed by separate eyes,
so there's no overlap.
For each eye we added a virtual periphery camera
that has a very wide field of view
but low output resolution.
We averaged the colors of each frame according
to a precomputed, distance-limited Voronoi diagram.
Each LED is then set to the average color
in the CIELAB color space, and we used that
because it averages the colors much nicer
than the other color spaces.
And the results, you can see, are shown on the right.
We then convert those colors
from the graphics card's sRGB color space
into the linear RGB color space used by the LEDs
and apply a per-channel calibration
that matches the peripheral LEDs
with the main display's color and brightness.
Finally we send those resulting colors to the Arduino,
using a serial connection, which then communicates
with the LEDs to set their colors.
The LEDs themselves are wired in series,
using a daisy-chained serial protocol
to reduce the number of control wires down to one,
so we don't need a giant mess of wires
to control all 100 LEDs.
Both of our prototypes update those LED strips
at over 75 frames per second,
which matches the update rate of the main display.
With the sparse periphery we can render content
independently of the main display.
Consequently we came up with three different visualizations
that can be applied to our sparse periphery.
First a designer can choose to simply render
the full environment to the periphery.
That's useful for simply providing extra context
to the user, treating this as an extension
of the main display.
Second, the sparse periphery can be used
to only render particular objects of interest
in the manner of a heads-up display.
Here, in one of our study tasks,
we render a particular target, that's here a white target,
into the periphery.
Finally, we developed a visualization technique
which we call the countervection visualization.
When the camera moves through controller input
or scripted events, we display a pattern that moves
in the same direction as the movement,
or, in other words, in the opposite direction
as the environment.
This helps to reduce the apparent motion perceived
by the peripheral vision.
Our goal with this visualization was
to reduce motion sickness by reducing vection.
The sparse periphery on its own already increases vection
if you render it using the traditional environment mode,
since it fills your peripheral vision
with moving targets, and initially we thought
to create a visualization to enhance that vection effect.
So we created a visualization where those bars move
in the opposite direction as the motion,
so in the same direction as the environment.
In this diagram you can see how that works
sort of diagrammatically.
If the user is moving backwards, for example,
then all the visual motion will actually be
in the forward direction in all of the zones.
This seems natural actually.
We wanted to enhance the user's immersion
by enhancing the sense of motion, but it turns out
that this is really, really nauseating.
We couldn't even stand it,
so we couldn't let our participants try this condition.
Surprisingly, reversing the flow of motion
to produce countervection significantly reduced feelings
of nausea in pilot testing, even when compared
with having no peripheral stimulation at all.
Here we show diagrammatically how that works.
If you're moving backwards in the visual scene,
then the periphery will move backwards,
and only the environment will move forwards.
We hypothesized that the reduction in vection is due
to conflicting information between the near periphery,
in red, and the far periphery, in blue.
But we haven't fully tested this,
and further research would be needed
to tease out exactly why this works.
We're not the only ones who tried to do some
motion sickness reduction through peripheral manipulation.
Fernandes and Feiner developed the technique
which shrinks the field of view gradually during motion,
and that works pretty well.
We're taking a different approach to try
to maintain the full field of view.
Our countervection visualization is implemented
as a series of blue and green stripes,
this is a bit of a confusing video to be honest,
which are rendered on a pair of spheres
only visible to the peripheral cameras.
The textures on the spheres are dynamically adjusted
to achieve the desired effect.
Now, we don't render the countervection visualization
if the user's only moving the head,
because head motion doesn't generate vection.
Only extrinsic motion does.
Here again you an see the countervection visualization
in action.
If the user's moving forward, you can see
the stripes are rendered moving forwards as well.
If the user turns, the stripes are rendered rotating
in the same direction as the motion.
And if the user stands still,
the regular field of view is rendered.
The regular environment is rendered.
We ran a user study to quantify the effects
of sparse periphery
on user performance and comfort within VR.
Our first study aimed to quantify the spatial awareness
afforded by the expanded periphery.
Users were asked to perform a very simple search task,
trying to locate a white target from among a grid
of green circles.
We ran this under three different conditions:
a narrow 50-degree field of view
that simulates typical augmented-reality environments,
a regular field of view
that uses the Oculus VR's entire range,
and a wide-field-of-view condition
where the target was rendered on the sparse periphery.
Each trial consisted of selecting a single target,
starting from the position of the previous target,
and because those were placed randomly, sometimes
in the field of view and sometimes not in the field of view,
we did not measure average trial completion time,
because it was too varied.
Instead we measured task performance
by the average targeting velocity.
That's dividing the distance to the next target
by the actual completion time in the trial.
Here higher velocities are better,
and we further subdivided by whether the target
was initially visible in the field of view or not.
And we found that in all cases
the sparse periphery provided a significant improvement
in targeting velocity, allowing users
to find and hit the target they wanted faster.
We also measured movement ratio,
which compares how much the user moves their head
compared with the theoretically optimal path.
So here lower ratios are actually better.
This metric measures the search overhead,
how much the user had to hunt around
to find the target of interest.
For both initially visible and initially invisible targets,
the augmented-reality condition was significantly worse
than the other two.
In our second study we sought to measure
the sparse periphery's effect on simulator sickness.
This study consisted of two different tasks.
In this first task users chased a little orb
as it moved around a fixed path through this forest scene,
using a gamepad controller to move themselves around.
We compared three different conditions here:
VR alone, so only the VR field of view,
VR with the periphery rendering a normal content,
and VR with the periphery rendering our
countervection visualization.
Users performed the first task three times,
once per condition, in random order.
And then, in the second task, users were allowed
to move themselves freely, using the gamepad,
and were allowed to also freely switch
between any of our three different conditions,
and they were asked to stay in that virtual environment
and move themselves around until they could
fully force-rank the conditions in order of comfort.
In total 14 of our participants completed both of the tasks.
In the second task, nine of the participants force-ranked
countervection first.
Two of the participants ranked
the ordinary-periphery condition first,
and three participants chose the standard
no-periphery condition.
In total it demonstrates that participants
generally preferred the sparse-periphery conditions,
and most preferred the countervection mode.
We feel that our work has opened up a number of avenues
for future exploration.
We'd like to further explore the application space
for sparse peripheral displays, for example using them
to provide unobtrusive peripheral notifications.
Our countervection visualization
provided some very interesting study results,
and we'd like to further push on that
to gain more insight into our technique.
Further research into this technique should help
to further mitigate simulator sickness.
Finally, we'd like to investigate
whether the sparseness itself helps
to reduce simulator sickness.
Preliminary results suggest that that effect may exist,
but a full comparison study, with full resolution periphery,
would be needed to really evaluate this.
OK. Thank you very much for listening.
(applause)