Hello everyone! I am Yaxuan Li, a master's student from McGill University.
It is my honor to present our work here.
Towards Context-aware Automatic Haptic Effect Generation for Home Theatre Environments.
4D theatre goes beyond 3D viewing, further enhancing the immersion of
the audience through the addition of physical effects such as haptic feedback.
To provide this 4D experience, cinema seats may include motion and
vibrotactile effects, bursts of air, mist effects, and thermal stimuli.
These are intended to enhance realism, increase enjoyment, and improve immersion.
Movies are rarely filmed with the haptic effects already included.
So, a separate, expensive, haptic authoring process is necessary.
This authoring needs to make sure that the haptic content matches the audiovisual content;
There are some tools that assist in this process,
such as H-Studio for kinematic and
vibrotactile effects and Mango for tactile effects,
but the authoring task remains difficult.
For the rest of this presentation, I will focus on vibrotactile authoring.
As an alternative to manual authoring, automatic haptic effects generation
methods have been developed to simplify the authoring process.
These methods typically produce haptic effects from either the audio or visual movie content.
The previous tools, however, only use a single modality, audio or visual, as input.
We propose to leverage both modalities to improve the generation of haptic effects.
To do so, we analyze audio cues of the soundtrack and calculate the salience of the video content.
As a starting point, we use the saliency map from the visual content to determine the
spatial location of the corresponding events.
These locations are then used to assign weights to the actuators embedded in the chair
to generate the spatiotemporal vibrotactile effects.
Psychoacoustic parameters extracted from the audio are used to determine vibration intensity.
Our approach generates haptic output that reinforces the high-level context of the movie.
From the audio signal, we extract four psychoacoustic measures of
sharpness, booming, low-frequency energy, and loudness.
Here's an example sound extracted from a movie clip.
We see the one high sharpness values corresponding to an explosion at time t =
4 and another high sharpness corresponding to engine sounds at t = 37.
Likewise, booming, roughness, and loudness values indicate when noteworthy events are happened.
Whenever these measures exceed a pre-set threshold, we render haptic effects proportional
to the amplitude of the corresponding psychoacoustic measure.
Then, we fused four parameters after thresholding on normalized psychoacoustic parameter values.
The normalized peak values are directly used as vibration amplitudes.
In a word, from audio, we extracted "when and how intense" cues.
Our algorithm fuses the outputs from the four measures,
resulting in various haptic effects being rendered in response to key auditory events in the movie,
such as banging, smashing, impact, and clattering.
In addition to extracting events from the audio signal,
we also analyze visual saliency using a CNN model.
We build the model according to VGG.
We added spatial attention module and channel-wise attention module
for low-level and high-level features, respectively.
This helps identify the location and size of the corresponding event
to activate spatially relevant actuators.
As shown here, with the left image as the input for the machine learning model,
we obtain the right image estimating the saliency area.
We estimated the saliency area for each frame of the video.
Together, the audio and visual analysis provides us with a rich characterization of events in the scene,
including their type, intensity, and location, and size on the screen.
Fusing these two streams of information results in more interesting and relevant
haptic output than is available from one stream alone.
We equipped a chair with nine vibrotactile actuators for the hardware platform:
five on the back, two on the seat, and two on the armrests.
This way, we try to maximize the area over the body to deliver haptic output.
As the movie clip plays, a microcontroller receives signals from the computer and
simultaneously triggers the actuators to render the vibrotactile effects.
To improve the spatiotemporal resolution of the haptic effects, we employ the technique of phantom sensations.
This works by simultaneously vibrating multiple adjacent actuators,
chosen from the vertices of the enclosing triangles,
with appropriate amplitudes to create the illusion of a vibration somewhere between them.
As an example of the phantom rendering, we can create a virtual vibration sensation at position
C12 by activating actuators A5 with a stronger amplitude than A6 and A9.
Further details are available in our paper.
Now let's take a look at the system in action.
Below the video clip, the visual attention detection is visualized on the left.
The auditory atmosphere analysis is shown in the middle,
and the vibration of the nine actuators is represented on the right.
When the mutants burst in and crash the glass door, high audio fusion parameters
are present, and actuators are triggered according to saliency on the frame.
We conducted a user study with 16 participants to evaluate the effectiveness of our haptic rendering algorithm.
This study used six short clips, ranging from sci-fi to action, cartoon, family-comedy, and horror movies.
The participants sat on the chair in front of a monitor to watch the six movie clips
with different renderings of haptic effects.
Here are the four haptic rendering conditions we used in the study:
Random, Audio, Visual, and Multimodal.
In the random haptic condition, vibration effects were generated randomly throughout the video clips.
In the audio condition, vibration effects were generated based on the audio parameter.
In the visual condition, the effects were determined according to the salient
area of the video clip and matched to the corresponding actuators.
In the multimodal condition, the full pipeline, including both audio and video inputs,
was used for generating vibration effects.
We gave a questionnaire to the participants which evaluated the movie watching
experience with haptic renderings on a 0-10 scale;
0 means "strongly disagree" to 10 for "strongly agree".
We asked for immersion, preference, harmony, and discomfort.
As a result, we observed a general trend that the Multimodal condition
haptic rendering through the whole pipeline,
outperforms all the other rendering methods such as audio only, visual only, or random.
The Random condition was always perceived to be the worst.
Note that, unlike the other measures, a low "Discomfort" rating is a better result.
Here are the overall results.
You can observe that the Multimodal condition earned the best scores in all the movie clips we used.
In other words, the results indicate that our multimodal haptic generation algorithm
provides the best hapto audiovisual experiences among the conditions.
The ANOVA results confirm that the rendering method has a significant effect on
all the performance ratings, with large effective sizes.
Although our preliminary results are promising, our system has some limitations,
especially in the situations of "no salient objects or regions" or "scene without sound".
This causes lacking information about the event that happened.
For example, the saliency analysis may not work on a dark, foggy scene with
horrifying sound in a horror movie, as depicted in the picture.
Likewise, the audio analysis may not work properly in the scenes like a
panorama without a dominant sound in a documentary movie.
Dealing with such limitations and expanding the algorithm to
multimodality would be our future work.
So far, we have introduced our automated haptic rendering algorithm and proved its effectiveness.
Now we discuss our algorithm's expandability.
Since the feature extraction pipeline is automated and provides many features,
the algorithm can simply be expandable by extracting more features
and matching them with other haptic modalities.
For example, we can extract color information to generate thermal effects or optical flow to make airflows.
The algorithm also has many venues in multimedia applications,
not only for movies but also VR, AR, and games.
We can extract scene data to figure out the context,
such as which item the user is interacting with,
the atmosphere or environment, etc., then automatically create haptic effects.
Here's our takeaway: we implemented an automatic haptic effect authoring algorithm,
which utilizes both audio and video cues and generates congruent spatiotemporal haptic effects
to the movie scene to enhance the user's movie watching experience.
Thank you for your attention.