Hi, everyone.
I'm Shaghayegh Roohi from Aalto University, Finland.
Today, I'm presenting Recognizing Emotional
Expression in Game Streams.
And you know that we cannot separate emotions from games.
Some study equates emotions with positive player experience.
Some others, no emotion as key factors in defining fun.
And also, we know that there is this public and underexplored
resource of game stream videos.
You can see here an example of the game stream
that we've used.
[VIDEO PLAYBACK]
- We're safe.
We can see that player how you show his emotion
and even narrates his feeling.
- We're safe.
[END PLAYBACK]
So we decided to investigate emotions inside a game stream
to see if we can detect the human-annotated emotional
events in those streams, and if we can identify the most
intense events of game streams.
We have contribution in this paper.
We have presented a new data set of emotional events
in game stream, which contains 17 videos around 2000
emotional events.
We also present a multimodal neural network for predicting--
for detecting these emotional events automatically,
which contain--
which uses facial expression, recognition, [INAUDIBLE]
sentiment, analysis, audio emotion analysis, and also
audio features like loudness and pitch.
For data sets, we use Unravel and Unravel 2 games, which
are a puzzle platformer games.
We selected these games because they are recently released
and there are multiple streams available for them.
And also, they have linear level designs,
so we expect that player's experience game events
in the same sequence.
And most of them have been praised
for being emotional engaging.
Our criteria for selection game stream
is that the face of a streamer should be
visible throughout the video.
A streamer should speak in English,
and only one person was playing and present during a stream.
Also, the automatic captioning is
available for the game stream.
This process resulted into 17 videos
by nine different streamers, two women and seven men.
Each video has been annotated by two annotators
into certain classes of emotions like happiness, amusement,
startle.
And also, each emotional event have
been labeled as being a top five event or not top five event.
We tried four types of classification.
The first one uses no event and event class.
The second one uses top five event and not top five events.
The third one uses four classes of no event,
pleasant, unpleasant, and neutral event,
pleasant event like happiness, amusement,
and unpleasant event like startle, frustration.
And neutral event like moved and wonder.
The last one uses no event class and all certain classes
of emotions available in the data set.
As I told every--
each video has been annotated by two annotators,
so we calculate their human agreement for each video.
And as you can see, the human agreement
decreases with more fine grained classes.
So, for example, you can see 14 classes
is not very reliable and with larger window length,
because we use a time window that
go along the whole video for predicting
this human agreement.
And with larger window length, the human agreement increases,
but we kind of lose temporal resolution.
Our automatic detection system, we
apply facial expression analysis on a streamer's face.
We apply audio analysis on a streamer's voice
and also sentiment analysis on streamer's speech.
All of these modules are trained on public large data set,
which outputs the probability of different basic emotions
or positivity of a speech.
I put up these modules or time set of signals.
We combined this time sending signals
in a time window using a temporal neural network, which
is trained on our smaller and custom data set and out--
this is neural network act will say
probability of emotional event.
Here, you can see an example of our input signal
as I told in the large time window, the kind
of loose temporal resolution.
So you can see that one annotator detected two events
in this time window, and another thing is
that you can see one annotator detected this event as startle,
and the other one detected it as surprise.
Because surprise and startle have similar expressions,
they are harder to be distinguished by humans.
And here is an example of a game stream alongside the input
signals.
(ON VIDEO) Hoo, hoo, hoo, hoo!
Hoo, hoo, hoo, hoo!
Here, you can see the results of our automatic and emotional
event detection.
We used a different combination of input signals,
and the first column shows just using facial expressions.
Second one, facial expression sentiment analysis.
Third one, facial expressions, sentiment
and audio emotion analysis.
Fourth one, facial expression, sentiment analysis,
and audio features like loudness and pitch.
The last one uses all input signals.
Here, again, like human agreement,
the performance decreases with more fine grained classes,
and you can see that window length of [INAUDIBLE] almost
always produces the worst result,
but we cannot see significant difference between other window
lengths.
At least, in our case, facial expression and audio features
was the most informative signals.
And their facial expressions sentiment analysis
and audio features produces the best results here.
But I think more input signals cannot help us every time.
[INAUDIBLE] maybe because of over fitting since our data set
is fairly small.
Unless the input signal is very informative,
adding more inputs, a signal can lead to over fitting.
And another thing that I want to mention
is that audio emotion analysis had good results on other data
sets.
You can find more details on the paper about that,
but the performance of it, [INAUDIBLE]
playing on game streams reduces.
And the reason is that because here,
besides a streamer's voice, we have game audio as well.
Here, you can see the percentage of missing events using
the best performance of two class model.
Our classification-- as our system is mostly--
has a good performance and then taking
positive emotional expression, and some subtle emotions
like moved, wonder, and puzzlement
have high false activity maybe because that they have
lower intensity expression.
And also, [INAUDIBLE] false activity for negative emotions
like frustration and startle maybe because that they are
immediately followed by a positive--
immediate positive expressions like smiling.
As future work, we want to add annotation of gameplay events
as an complementary material.
We want to investigate the effect
of differences in gender, age, and culture on our system.
And also we want to test our system on play test videos
without game audio.
In summary, we presented a new data set of emotional events
in game streams.
We have presented an automated detection of emotional events
in game stream using multimodal neural networks.
And we showed that identifying emotion
is the hard test for both human and artificial neural networks.
But with a limited number of classes,
we can get reliable results.
For example, in case of just detecting events,
we get automatic detection accuracy
of 70% which is 70% on par with human argument of 68%.
And this suggests applications in highlight--
video highlight detection and pre-screening videos
for later analysis.
Thank you.
[APPLAUSE]
So ladies are open to questions.
Here's one.
Thank you.
Very nice paper.
Nice analysis of the results.
Thank you.
And I like this progression from 2 class to 14 class results.
I see the challenges.
But I have a question basically about what's
on here, automatic detection accuracy of 70% comparing
to the human inter-rater agreement.
In the big table, we also saw 80% accuracy for the 2 class
to window, another 3 class to window.
Yeah.
Anyways, how do we interpret 70% accuracy
in case of 86% human inter-rater reliability?
Because that's not entirely evident to me
what it means to have an accuracy of 70% now.
I shouldn't-- that's a good question, thanks.
Human agreements cannot completely--
it's not completely comparable with automatic detection
analysis.
Since in the case of human arguments,
one person tries to predict another person.
But here, we kind of try to predict
the average of annotations of players
because of that they are not completely comparable.
Yeah.
But the 70% seems to assume some kind of reference
to a ground truth.
But a 68% human inter-rater agreement kind of
suggests that we don't really have a ground truth.
So shouldn't you need to do an extra layer of analysis
taking automatic detection back to humans
and say how meaningful and recognizable
is the outcome of my automatic classification
afterwards for, again, for human assessors?
Yeah, actually that would be interesting to investigate.
Yes, I guess.
I guess--
[INTERPOSING VOICES]
--but I don't know.
I mean, it's not meant as a critique but a suggestion.
Thank you.
For other questions?
Yes.
[INAUDIBLE]
Yes.
Hi.
[INAUDIBLE] electronic arts.
Thank you so much for the talk.
I'm really looking forward to reading this paper.
Could you-- I'm sure it's in the paper,
but could you describe a little bit
the setup and how you prompted your participants so
that they emoted so strongly as they did in the videos?
You mean--
Oh, these were game streams from--
and then where did you get them?
I hope you--
YouTube.
Oh, OK.
All of them.
Well, in that case, don't you think
there's a little bit of a performative aspect to--
Yeah, sure.
There is, I guess--
we have this problem with game streams.
They kind of exaggerate their feelings,
but we can apply this method on a video play
test that are more realistic.
But even in case of game streams,
it is good that they conduct narrate their feelings
because they want to attract their viewers.
And because of that, for example--
lets see the gaming face and this stuff because of that.
I guess, it's worse analyzing.
Yeah.
Thanks.
Thank you.
Time for one more question?
Let's see.
Hi.
[INAUDIBLE] Maybe I missed this, but did you train
the classifier for each specific video feed source, or--
Hmm, no.
--is it just the same classifier for all?
For all.
OK.
OK, so let's thank [INAUDIBLE] again for--
[APPLAUSE]
[INAUDIBLE] --and we stay [INAUDIBLE]
with the next presentation.
And to spice it up a bit, I [INAUDIBLE] this [INAUDIBLE]
for video.
For this paper, so you didn't expect that.
No.
[LAUGH]
Thank you very much.
[APPLAUSE]