Hello, my name is Jeremy and I’ll be presenting the paper “View-dependent Effects for 360
degree Virtual reality Video”.
This is part of the research I did when I was an intern at Adobe with Stephen, Cuong
and Daniel.
First, what is a view-dependent effect.
A View-dependent effect is an effect where the underlying parameters of the effect changes
depending on where the user looks, making it dynamic and real-time.
While a per-pixel effect is an effect that is directly encoded into the pixel, rendering
it static and offline.
This can be seen in standard video editors, where an effect, like a vignette, is directly
encoded in the video which causes noticeable gradients on the equirectangular frame for
a particular view.
This can be seen in the video here, where a drastic color change happens across the
frame, at the effect location.
While view-dependent effects are real-time, so a set of effect parameters can be interpolated
and applied to the current view the user has into the scene.
This allows us to remove unwanted gradients and uniformly apply the effect.
This can be seen here, where multiple effects, like tint, saturation, and contrast are applied
to the view you see on the screen.
We accomplish this by interpolating a sparse set of spatial keyframes that are embedded
into a 3-dimensional video space.
This includes a 2-dimensional location on the equirectangular frame and one temporal
dimension.
Each spatial keyframe then samples the parameter space at those particular coordinates.
Our interpolation works by treating each spatial point (for example S1, S2, and S3 in the figures
on the screen) as a radial basis functions (RBF).
We then use these to find the optimal weights so that we can smoothly interpolate between
each spatial keypoint depending on where the current view (v) is located.
In order to facilitate real-time rendering of effects, we built a video player for windows
and iOS.
The player takes in a set of spatial keyframes, determines their optimal weights for interpolation,
and renders those effect parameters onto the current view of the user in real-time.
For our prototype player, we implemented a set of standard filmic effects that include,
Blur (which is located on the left), tint (located in the middle), exposure (located
on the right)
Saturation (which is on the left), contrast (in the center), and finally vignette (which
is on the right)
With these 6 effects, we can use them for not only post-production and narrative driven
purposes, but they can also be used for Motion sickness (which you can see on the
left), by applying view-dependent vignettes to restrict the user’s view in the scene.
And Gaze guidance, (which is located on the right), by darkening the scene to redirect
the user’s view back to the primary point of interest in the video.
To facilitate the authoring of view-dependent effects.
We built an in-headset VR editor that allows a videographer or video professional with
the ability to apply view-dependent effects to 360 degree video.
One hand controls the application of effects, while another hand controls the video playback
and timeline.
Here we can see the 6 effects (blur, tint, vignette, exposure, contrast, and saturation)
and their corresponding effect panels for fine-tuning and manipulation.
The other hand controls the timeline and visualization of the effects.
The effect tracks (which is located on the left) visualizes the effect parameters over
time for the current view of the user.
The effect overlay (on the right) visualizes the effect parameters spatially across the
equirectangular frame for the current timestamp in the video.
Together, these provide a way for the user to see how an effect propagates spatially
across the frame and temporal in time.
These are all updated in real-time as the user drops spatial keyframe onto the video,
giving them real-time feedback for iteration and modification.
To evaluate the feasibility of view-dependent effects.
We conducted two studies, one was with a group of export user and the other with a general
audience.
In the first study, we recruited 8 experts who all have work with 360 degree video professionally.
After initial onboarding, thier task was to apply view-dependent effects to a 360 degree
video of a rocket launch.
Here, on the screen, you can see how the unique color styles for participants 2 and 5 were
applied to the video.
Also note that these are all shot with our player in VR.
The next experiment looked at the perception of view-dependent effects with a general audience.
Each participant was shown a 360 degree video while an experimenter switched between the
view-dependent effect and the regular video.
One video form the study can be seen here, which switches between a static and dynamic
view-dependent rendering of the scene.
That concludes the presentation for View-dependent effects.
Please see the paper for more details.
or contact me if you have any questions.