Hi, my name is Dae Ha Kim, and the title of our paper is hidden emotion detection using multi-modal signals.
First, introduction and problem statement.
Multi-modal emotion recognition is to grasp person’s emotions using various types of signals.
Visual signal from facial region and physiological signals of EEG and PNS are mainly used.
A representative method is to predict the visual and EEG signals
annotated on Russell's emotional model consisting of two-dimensional axes of Arousal and Valence.
There are two directions for multi-modal emotion recognition.
First, understand correlated emotional patterns through visual and physiological signals,
but it cannot deal with mismatch emotions.
Second, understand when person’s facial expressions and inner emotions are different
that is mismatch negativity, shortly MMN.
Until now, the latest signal analysis tools such as CNN have not been used for mismatch negativity.
In other words,
multi-modal emotion recognition mentioned before can be divided into matched emotional analysis based on CNN
and MMN analysis based on ERP signals.
Let’s move to the proposed method.
Our goal is to define various HED cases and validate hidden emotions using multi-modal signals.
The overall framework consists of three steps as follows.
In training phase, 2D CNN for visual image and LSTM for EEG sequence is frozen
and only parameters of fusion module is learned
to minimize ground truth and emotion guidance value, that is the output of fusion module.
In testing phase, 2D CNN and LSTM is basically used for extracting feature vectors
and get a fusion score through adaptive fusion.
At this time, we can utilize trained fusion module for weighting EEG feature vectors
when correlation of two signals is relatively small.
We empirically found that a clear fusion result can be obtained
when EEG signal is amplified.
Decision rule-based EEG signal amplification can be one of solution as shown in below figure.
However, if the image and EEG signals have the valence values of opposite sign,
the rule-based adaptive fusion (shortly AF) can produce an ambiguous result.
So, we propose neural network-based fusion module to capture various mismatch cases.
The loss function of fusion module has two terms:
RMSE loss function and L2 regularization.
The details of experimental results.
We filmed 23 participants and collected various mismatch cases.
We labeled HED dataset using self-assessment rule based on Russell’s emotional model.
Also, press sensor button is used for checking the subject’s actual feelings.
We achieved about 83% detection performance on HED evaluation dataset.
Fusion module is important to capture severe mismatch emotions,
such as fear and sad.
Also, we found that the expression of sad that could not be captured by AF alone was detected through the fusion module.
The surprising result was confirmed that the actual emotion (yellow part) and the fusion results (blue part) actually matched.
Even in happy emotion, we can see that the multi-modal results capture actual emotion well.
Additional qualitative results are shown below figures.
This study shows that the HED framework can be used in a variety of applications
such as lie detection and circumstance of jury witness.
These are references of the presentation.
Thanks!