Hello everyone, my name is Tong Xue and I’m a fourth year PhD student
at Beijing Institute of Technology in Beijing, China.
This is a presentation video of our CHI 2021 work
Titled: RCEA-360VR: Real-time, Continuous Emotion Annotation in 360° VR Videos
for Collecting Precise Viewport-dependent Ground Truth Labels.
This is a collaboration work with my co-authors in CWI, and TU Delft.
Watching 360° videos using HMD can provide interactive and immersive VR experience.
Whether the goal is to induce emotion or track and recognize emotion within VR,
it is important to collect accurate and precise ground truth emotion labels
However, unlike 2D videos, by allowing users to freely rotate their heads
and construct their own viewing experience, we can no longer be sure users’ annotations
pertain to that specific scene at any given point in time.
In general, emotion data collection takes place by post-stimuli self-reports,
like Self-assessment Manikin (SAM).
However, such self-reports are temporally imprecise,
since one can experience multiple emotions while watching video content.
Moreover, collecting emotion in VR can be time consuming, demand cognitive effort,
and even break the sense of presence and immersion.
To minimize such workload and distraction, we designed two joystick-based continuous
emotion annotation techniques, HaloLight and DotSize in our prior work.
The color of the visualization represents which quadrant they are annotating,
and the transparency or size means emotion intensity.
In this work, we first present and evaluate the usability
of RCEA-360VR system.
Our experiment was conducted in a controlled, indoor environment
with 32 participants using both HaloLight and DotSize.
They viewed eight 360° video clips through an HTC Vive Pro Eye HMD,
and they were instructed to annotate emotion states
by moving the joystick head into one of the four quadrants.
A helper function was embedded for easy lookup.
Besides head movement, annotation data, and physiological signals through the Empatica
E4 band.
Participants were also asked to report their overall emotions
using a within-VR SAM rating scale and fill in SSQ,
IPQ, and NASA-TLX questionnaires.
Through our controlled usability evaluation, we found no significant differences
between HaloLight and DotSize concerning motion sickness, presence,
or helper function usage amount.
For mental workload, we do not find significant differences
neither from NASA-TLX nor physiological measures among HaloLight, DotSize and the within-VR
SAM ratings.
Together, our findings indicate HaloLight and DotSize
don’t result in high sickness, workload, nor break presence.
Then we verified the reliability of our continuous annotations.
The mean Valence and Arousal (V-A) ratings for HaloLight and DotSize
are in line with ratings from original dataset, as well as our within-VR SAM ratings.
Also, V-A are rated similarly across HaloLight and DotSize.
To ensure our continuous annotations are effective for building precise ground truth labels,
we develop a segment-level viewport-dependent annotation fusion method.
We first calculate time delays and did continuous annotation time-alignment by
DTW.
Then we run hierarchical clustering to cluster users based on similarities in
their viewing behavior.
Lastly, we conducted frame-level and segment-level fusion approaches
to robustly fuse multiple viewport-dependent annotations
These fusion results provide temporal details into the
peaks, valleys,
and trends of users’ experience, which enable a more fine-grained temporal
lens to understand emotion states and specific
scenes of an immersive 360° viewing experience.
Thank you for watching this video!
All our code is open source and can be obtained by scanning QR code or visiting GitHub.
If you have any questions or want to know more details about our RCEA-360VR work,
please feel free to contact us!