Hello everyone, I'm Yu-Hsin Lin from National Taiwan University.
Today i'm going to present HapticSeer.
Haptic feedback could significantly enhance gaming experience, which is a popular virtual activity.
There are so many games in the world! According to a well-known review aggregator Metacritic,
there are more than 17000 commercial games around the world; moreover, there are more
than 500 games scored above 90.
That is to say,
there are tons of high quality content with great potentials for haptic developers to use.
However, adding haptics to a commercial game usually
requires having access to modify its codebase. This limitation makes it challenging to add
haptics to commercial games, which are typically close-sourced. One workaround that haptic developers
have used is the in-game audio analysis based on audio features such as volume and impulses.
For instance, HaptiVec, a controller presented in CHI '19 adjusts its operating frequency by the
volume of a rainfall sound playing in the background to create a "haptic rain".
However, this audio-only approaches is limited in precision and limited in the types of events it can support.
Therefore, we present HapticSeer, a multi-channel black-box, platform-agnostic approach to detecting
game events for real-time haptic feedback.
Our system is motivated by two key insights:
On the one hand, all games produce three types of data streams, video, audio and controller I/O, which
can be analyzed to detect game events.
On the other hand, according to our survey of top FPS (First-person Shooter) and racing games,
most games with the same genre use a small number of user interface design patterns so that
code written to detect events for a game can be reused for many other games.
To be more specific, a haptic developer needs to replace an unsuitable component only with one
tailored for the target game.
They don't need to swap all the components in this scenario.
We developed an open source framework to capture video, audio, and control I/O in real time.
It provides the capture data, event system and abstraction for feature extractors and event detectors to simplify haptic devices' integration.
To validate the system's correctness and performance,
We implemented several real-time multi-channel event detectors for commercial PC and VR (virtual reality) driving and first-person shooter games.
For example, we use a pulse extractor alone to verify that HapticSeer could achieve 100% accuracy in predictable conditions,
showing that the audio approach is reliable.
Moreover, we evaluated the system in natural settings for reaching a better understanding
of HapticSeer's capability while being used as a developer tool.
For instance, we launched an OCR (Optical Character Recognition) number extractor and an Xbox controller input capturer to support an inertial detector and a
speed detector for a famous racing simulator Project CARS 2.
Then compare their output with ground-truth extracted from the game's telemetry tool.
Furthermore, to collect feedback from haptic developers, we invited several developers to integrate their haptic devices with commercial games
using HapticSeer.
Developers reported that HapticSeer could make the integration of haptic devices and commercial games easier
We discussed some future work in the paper.
For example, user reported that it will be easier to assemble components if HapticSeer has a graphical user interface.
Furthermore, we plan to build a toolkit help users create and share their components at ease, since
the power of an open platform will prevail after building a solid userbase.
In conclusion, we have shown the feasibility of extracting game events from commercial games to integrate haptic devices
by developing an open source framework making it easier for haptic designers to support games
even without accesses to source code or a specific API (Application Programming Interface)
With HapticSeer's modular and flexible design we hope that it could form an open source community and enable swift integration
of haptic developers works into more commercial games.
Thanks my co-author and collaborators for all their hard work on this project!
Please find more details for both implementation and usage instruction in our paper and check the source code published on Github. Thank you very much.