Hello, everyone!
I’m Gyeore Yun from POSTECH in South Korea.
I'm here to talk about my research, which focuses on creating real-time, selective,
and multimodal haptic effects from sound to enhance the gaming experience.
Before I begin my presentation, this research was supported by Google.
So, let's get started!
There is a game called Astro's Playroom on Playstation 5.
have any of you played this game?
If you have, then you may experienced the amazing haptic effects of the latest DualSense
controller.
This game showcases the controller's features by using sound to create diverse and rich
haptic experiences that are absolutely mind-blowing.
In the game, sounds are usually converted into haptic effects.
However, not all sound events are converted into haptic effects.
For example, if you’re in a peaceful situation in the game with wave sounds of gentle water,
and character dialogues, you will not get any haptic effects.
Why?
Simply, these sounds are not typically associated with tactile sensations in real life.
On the other hand, when something is hit or struck, you can feel the haptic effects.
We can expect both sound and haptic effects from such events in real life.
This game does a great job of providing haptic effects selectively.
Also, we can experience realistic and immersive haptic effects generated from sound signals.
But, there are limitations to just using vibration to simulate hitting sensations.
Imagine how much more immersive the game would be if you could feel like you were actually
hitting something.
These sensations can be simulated by using impact actuators.
Anyway, games that provide such amazing haptic effects typically have sound events that are
marked for haptic effects in advance.
However, many other games we usually play are created without considering this.
That's where our research comes in.
We're developing a real-time system that can selectively provide sound-based haptic effects,
even for games that are not specifically designed for haptic effects.
The effects will be experienced vividly using impacts and vibrations, called multimodal
haptic effects.
A real-time system like this is critical in terms of delay.
Let's consider the situation where the computer outputs a punching sound that requires haptic
effects.
Impulsive sounds like this usually have very short attack and a maximum loudness peak,
as well as a release state of residual sound.
I have represented this with a triangle.
And, Let's consider this yellow one is the current point.
Then, there is a very short delay in reproducing these sounds through a speaker.
We need to determine whether the currently playing sound has an appropriate haptic effect.
Since we cannot make a judgment beforehand, there is a delay in capturing meaningful sound
information.
Let’s call it data capture delay, and it largely depends on the capture length.
Additionally, it will take computational time to make a judgment from the captured information
whether a haptic effect is suitable or not, and this computational delay is inevitable.
If the system decides that the haptic effect is appropriate, then we need to drive the
haptic actuator.
We can't ignore the delay caused by physically moving the mass of the actuator, called haptic
generation delay.
Now, the system has a delay between the sound we hear and the haptic effect.
We conducted research by minimizing data capture delay and computational delay, excluding the
haptic generation delay, which depends on the hardware.
So, let’s briefly see how our system works.
Our system captures sound streams in a sliding window manner, where each unit of time captures
a fixed length of sound data.
We then analyze each captured sound data independently and provide haptic effects accordingly.
To make a judgment, specific features are extracted from each window.
We divide the window into three sub-windows to consider temporal characteristics, and
we extract log-energy, MFCC, and spectral features from each sub-window to compose 54-dimensional
features from a single window.
A random forest classifier, one of the classical machine learning models, selects the most
appropriate haptic effect for the current window.
It chooses one of the following: no haptic effect, vibration, impact, or vibration and
impact simultaneously.
To make this possible, we collected data ourselves.
We created datasets of role-playing games and first-person shooter games.
Earlier, we mentioned that the sounds we want to capture generally have loudness peaks.
We extracted peaks from the sounds in many PRG and FPS gameplay videos on YouTube and
labeled which haptic effect was appropriate by listening to them ourselves.
So we have the dataset and all we have to do is just train the random forest model and
optimize it.
We set the number of trees in the random forest, named number of Bag, and the capture window
length as parameters to optimize the classifier.
Both have a trade-off between the accuracy of the detector and the sound-to-haptic delay.
In conclusion, we found the optimal parameters that show the highest accuracy while achieving
real-time performance.
Based on the resulted class, the sound-based vibration or an impact effect is generated
from each actuator.
We used low-pass filtering for vibration and the impact with the maximum amplitude.
For more details, please see our paper.
To render it, we attached actuators for impact and vibration to the mouse, the most familiar
game controller.
All of these are commercial products, and the impact provides a stimulus that feels
like hitting once, like in the video.
We evaluated our system through a user study.
To provide users with the same environment regardless of game difficulty, we assumed
gameplay scenarios while showing gameplay videos.
20 participants were recruited for RPG and FPS games, respectively.
As a control, we used low-pass filtered vibration and a state-of-the-art algorithm based on
psychoacoustics.
We prepared two conditions of our system with vibration only and both vibration and impact.
Users watched the gameplay videos with real-time haptic effects and responded to six user experience
items on a 100-point scale.
In conclusion, we confirmed that our system can provide better gaming experiences than
the other systems.
However, there was no significant difference between using vibration only and using both
vibration and impact, and we inferred from the interviews that this result is due to
personal preference differences.
Thank you for your interest in my presentation.