hello everyone my name is yousef and
today i'll be talking to you about
automatic frustration detection using
thermal imaging
during our daily lives humans interact
and have to analyze a lot of social
signals
even for humans this can be a hard task
let alone robots
so for example if we look at the photo
on the right there the person in the
middle have a facial expression that a
robot or a machine learning algorithm
would analyze as sadness
although as humans we can easily tell
that this is not the case
that's why in robots we need to use more
multimodal systems
to understand more about human social
signals
especially when it comes to internal
states
internal states are a bit harder to
understand as they describe the internal
state of the person instead of a visual
state
internal states are important because
they help us understand more about
humans
and it makes our perception systems more
robust
it has been established in the
literature that frustration is one of
the most common effective states humans
go through when interacting with robots
frustration can cause lower productivity
lower learning efficiency and slower
response time that's why in this
research we focus more on detecting
frustration
a lot of the previous literature
discussed how to detect effective states
effective states like stress and
cognitive load frustration and
disappointment
however
a lot of these use vision or other
physiological sensors that can be
intrusive or inaccurate
therefore in our research we're getting
we're going to use thermal imaging
instead
thermal imaging has been discussed a lot
in psychology literature especially when
it comes to thermal regions
it has been discussed that each region
on our faces can correlate to a
different emotion or a different
effective state
hence in this
paper we've used the similar regions
like the forehead the cheek the nose and
the lower lip
so our goal is to use thermal imaging to
detect frustration
and we're going to do that by comparing
the model performances using rgb facial
thermal and electrodema activity
features
added to that we're using a sequential
feature selector to understand more
about what features can correlate to the
best
performance of the model
we've decided to collect data from 18
participants aged between 21 and 39
years old
the participants went through two tasks
one cognitive load-induced frustration
task and the other failure-induced
frustration task
between each task the person was given a
nasa tlx questionnaire to know how
frustrated the person was
in the cognitive load and used
frustration the person had to program
the robot to go from the living room to
the hallway using the block diagrams as
seen on the figure
this programming task has been adopted
from previous works
added to that the person was hearing a
buzzer
when the buzzer goes off the person had
to change to a mental rotation task and
then go back to the main programming
task
in the failure induced frustration task
the person had to play a trivia game
that is biased which the robot would
tell the participant that their answer
is wrong even though it's completely
correct
[Music]
in our system we used both rgb and
thermal cameras and they were both
calibrated
in addition we've used electrodermal
activities
and extracted the tonic and phasic
components out of each data point the
full architecture of the system is shown
in the figure
the system is ros-based which means that
the whole pipeline is connected through
roughs and synced
and then features are extracted through
it
open face was used to extract the
landmarks on the facial data
and then correlated with the thermal
data to determine the regions due to the
landmarks
in addition each event
during the robot interaction was
automatically logged through ross
we use the k-nearest neighbor model to
classify if the person is being
frustrated or not
we have four models
with that were trained on thermal rgb
and eda
separately and together
in addition the ground truth data was
labeled based on both the cognitive and
the failure induced frustration tasks
in the cognitive load induced tasks
the whole task was taken as a
frustration instance
on the other hand in the fair and use
task only when the person answered
correctly and the robot replied
incorrect
seven seconds afterwards it was labeled
as a frustration instance
also we've used the f1 and accuracy as
metrics to understand more about the
performance of the model
it can be seen here on the figure that
the x-axis is the three different window
sizes that we've used and trained the
models on
and on the y-axis there is the accuracy
of the model and the two different tasks
that we've used is the cognitive fail
it can be seen here if we look at the
thermal data that the accuracy peaks on
the 7 second window
this means that the changes in the
thermal data is a lot slower compared to
the rgb data
on the rgb accuracy it peaks on the 3.5
second window
which means that
one second is too little data and seven
seconds is too much data
so it leads to less accuracy
in conclusion comparing the thermal and
the rgb features
in the one second and the seven second
windows
the accuracies are very similar
that means that thermal data can achieve
similar accuracies
in these windows
we've also used the sequential floating
feature selector to select the highly
ranked features that would affect the
performance of the model
and it can be seen here that the cheek
region was completely discarded from the
data
nose forehead and lower lip were the
main ones that were chosen
for the rgb modality the action units
that were extracted
the main ones were actually 28
2 4 and 7.
these action units correlated to
effective states that were chosen from
the literature
to check the full list of chosen action
units please check our publication
in this work we've extracted facial
thermal regions by calibrating both the
rgb and the thermal imaging we've also
compared three different modalities by
training three different models on each
modality the rgb the thermal and the eda
modalities we've also selected the most
prominent features
on by using the sequential feature
selector
our results show that the thermal
changes are less drastic than the rgb
as it was it needed seven seconds to
reach the maximum accuracy
added to that the cheek as a thermal
region is less relevant for detecting
frustration as it was completely
discarded by the feature selector
and just using thermal features can
reach similar accuracies to just using
the rgb features or just action units
in our future work we're going to focus
more on frustration in the wild as it
can be very dynamic
also we're going to deploy our system
using gamified therapy
hence using our systems in healthcare
both branches will help us explore more
about the models that we can use to
understand about the temporal aspect of
the data especially when it comes to
frustration in the wild
thank you so much for listening to my
video and let me know if you have any
questions in the main session