Hello! My name is Paulo Bala and I'm presenting
"Staying on Track: a Comparative Study on the Use of Optical Flow in 360ยบ Video to Mitigate Visually Induced Motion Sickness".
I'll start off with some key concepts.
VIMS or Visually Induced Motion Sickness is sometimes referred as VR sickness
or cybersickness and occurs strictly from visual stimulation.
The Simulator Sickness Questionnaire, SSQ, is used to assess and differentiate VIMS,
whose main symptom category is disorientation, from motion sickness and
non virtual simulator sickness. VIMS is polsymptomatic, meaning that there is
multiple symptoms, and polygenic, meaning that those symptoms manifest
differently across individuals. Furthermore, certain set of subpopulations
are more susceptible. Previous work offers multiple solutions to prevent
VIMS such as non natural travel interfaces like teleportation, walking
based techniques like walking in place and redirected walking and visual
optimizations. We are particularly interested in the latter since they are
applicable to 360 video and will expand on two of such techniques.
The first is Independent Visual Background, IVB, where visual elements consistent
with the real world serve as anchors when introduced into the virtual world.
In previous works, IVBs would use grids, horizons, fake noses, clouds, among others.
The second is Field of View Manipulation. Previous work proved that
wider fields of view caused more symptoms since users are more exposed to
the visual periphery, an area that is more sensitive to motion. A solution to
this has been the inclusion of the vignettes to block this area, restricting the field
of view, as done by Fernandes and Feiner, where the movement speed of the player
was matched to the radius of the field of view. Motivated by this last work and
applying it to 360 video, we wanted to explore the use of optical flow, a motion
pattern caused by the relative motion of objects in the visual scene compared to
the camera or the observer, and we asked two high-level questions: how can we use
the optical flow of 360 Video to mitigate VIMS?
How can we apply and combine techniques that mitigate films in VR to 360 video?
Our goal by merging optical flow and VIMS prevention techniques is to have a
technique that is only present when is needed, maintaining immersion in the
medium. To answer these questions, we started by choosing a popular 360 video
of a roller coaster with considerable motion. The video chosen is present on
two datasets of saliency maps and using satellite images and the original
video, we segmented the track contour according to the track features like a
"launch track", "double-dip", "Helixes" and "headchoppers". We process the video with a
Gunnar Farneback algorithm, a dense optical flow algorithm, and exported X and Y
optical flow vectors to a CSV file. To reduce file size and make it operable
for runtime use, we grouped optical flow vectors every 5 frames and in 5 by 5
pixel squares. We mapped optical flow to time and used it to complete the table
with a description of optical flow patterns and peaks. Based on
the track description, we isolate two segments of interests: S2, in blue,
where the wagons are starting to gain speed and S3, in yellow, when the wagons
are at max speeds causing drastic changes in the optical flow.
During runtime, the optical flow values of the area where the user is looking at are
aggregated in a magnitude vector.This magnitude vector is used to determine
the size of the field of view. A high optical flow, meaning movement, requires a
small field of view to limit the visual periphery. A low optical flow, meaning
that the environment is stable, allows for a larger field of view.
The independent visual background and restricted field of view techniques were
based on the "tunneling demo" from the Google Dream Elements and implemented
in Unity. Here you can see a simulation of this behavior in the unity editor.
A Gear VR with a Samsung s6 was used for the study. Based on this implementation
we organized a user study. The study used a factorial design with two
independent variables, IVB and restricted field of view, resulting in
four conditions, as you can see on the right side. 24 participants experienced
the four conditions in a random order over a period of two weeks with at least
a day between sessions. Prior to the study, participants reported
demographic data, experience with VR and filled out the motion sickness
susceptibility questionnaire short form. Our population was mostly without
experience with VR and presented the mean MSSQ raw score below the general population.
Besides analyzing the full population
we repeated all analysis with two subpopulations.
This includes a female subpopulation
that previous work has shown is more susceptible to VIMS
and a subpopulation made of the upper 75 percentile of the MSSQ-short.
For each session, participants fill out a Igroup Presence questionnaire,
a Simulator Sickness Questionnaire, before and after the session, a Visual Questionnaire,
similar to the one applied in Fernandez and Feiner's work. We also included an
optional comment field if the participants wanted to report problems
with their experience. Finally, Head direction data from each session was
recorded and decomposed into measures like yaw, roll, pitch, latitude and longitude.
Taking into account the track segments, we also looked at distance,
meaning how much they move between readings, and distance to point of interest,
meaning the distance between the head direction and the point of interest.
The lower the value, the more the participant was looking at the point of interest.
In our case, the point of interest was the
rollercoaster track and this was based on the dataset saliency maps of this video.
Focusing now on the results, analysis was done in R and all data was
nonparametric with the exception of spatial presence and experienced realism.
There was no statistical significant difference between conditions but the
distribution of disorientation, nausea and oculomotor
is consistent with VIMS. The boxplot on the right represents SSQ total score
before and after the session and the difference between them.
Ideally, the latter should be a narrow box plot with the median at zero meaning that the
condition did not cause an increase in symptoms. From our conditions, using only
a restricted field of view, in blue, seems to be the best alternative in reducing VIMS.
In terms of Presence, the results are expected since using VIMS prevention
techniques introduce noticeable visual interferences. Conditions with restricted
field of view were shown to have lower scores for general presence and spatial presence,
as in a way these techniques are more "aggressive" and they hide more
of the visual periphery. From the visual statements, participants were aware of
the visual changes made in each condition. For brevity, we only
highlight some of the comments made. Some participants thought that the grid was a
mistake, with P10 reporting to "be more careful with the blue lines around the
visual field". Some participants dislike the rate of change of the field of view
and finally a participant reported that "Some of the parts where the field of
view got smaller were really strange to experience, my brain kept wanting to
focus on the yellow track". So is this behaviour exclusive to this condition?
Looking at this small example from S2A and S3B, participants use the
slow movement of S2 to explore the environment but during the double
helixes of S3B, they are focused on the track. Looking at the head direction axes,
there is similar behaviour between conditions but there is a different
overall behaviour when we consider the segments. In the blue segments of S2
there is more exploration either by looking side to side or up and down.
When the yellow segments of s3 start, the more turbulent part of the ride,
the participants focus on the point of interest, in this case, the yellow track.
This type of behaviour is consistent in the other segments and
from the hotspots, we can see that there is not a notable variation between the conditions.
While we cannot compare S2 and S3 statistically since they
have different time windows, we can compare conditions in each of the segments.
For the full population the distance and the distance to point of interest
are similar among conditions and no statistical significance was found.
However, looking at the females subpopulation, in the absence of independent
visual background there was more movement in S2 and S3 and participants
were less focused on the point of interest in S3.
For the upper 75th percentile subpopulation,
a main interaction effect was found where adding an independent visual
background to a restricted field of view resulted in less movement in S3.
Returning to our research questions, optical flow can help to contextualize
the video like for example, the small wave pattern in S2B corresponds to
equally distant light poles.
And optical flow can also help identify
extreme motion like double helixes in S3B.
Conceptually, we can use optical flow
in 360 Video to mitigate VIMS but we also need to take into account
the media, for example, the existence of singular or multiple points of interest,
the existence of a stationary element like the
roller-coaster wagon that offers consistent low optical flow, and in
general how optical flow is distributed in time and space.
Furthermore understanding user behavior is crucial since by staying on track, the use of the
optical flow as a control mechanism for the techniques was not optimal.
The "staying on track" behavior might have various causes like participants having
prior experience of a rollercoaster and knowing the behavior to adopt or that
participants know the direction they are headed since it's an on-rails experience
or that the interplay between the optical flow and the techniques is
guiding the participants to focus on the yellow track. Regardless, these two
components suggests that rollercoasters might not
be the perfect case study for VR Sickness studies and further studies could be
carried out where ideally the media would have an unknown and variable path
of motion, with considerable optical flow and a longer duration.
Finally, our second research question,
how we can apply or combine techniques that mitigate VIMS in VR to 360 video?
As expected using techniques that introduce non-realistic
elements are bound to affect Presence components. In our case, the more
noticeable technique, restricted field of view, affected general presence and spatial
presence and involvement. No statistical significance on simulator sickness
questionnaire for the full population was found but the restricted
field of view not only produced the lowest simulator sickness questionnaire
scores but was well received in terms of comfort, enjoyability and desire.
One of our takeaways is to not only focus on the SSQ components but to look at the
objective measures like distance and distance to point of interest as
measures to understand and contextualize behaviours especially when applied
to our subpopulations. In conclusion, with this paper we presented a two-way
repeated measure studies with 24 participants using independent visual
background and the restricted field of view, controlled by an optical flow of a 360 video.
Using a fusion of quantitative data and knowledge of the media and the
knowledge of the user behaviour, we were able to understand if and when these
techniques worked, pointing out improvements needed for future studies
and for generalization. Thank you and please check out our paper and
supplementary materials for more information on this work.