Hello, my name is Shu Liu. I am honoured to present our work on The Empathetic Car:
Exploring Emotion Inference via Driver Behaviour and Traffic Context.
Driving is an integral part of modern life. In Germany for example, 68% of the working
population uses their car for commuting. However, daily commute with car is one
of the major source of negative emotions such as stress and anger.
The accumulated negative emotions do not only have bad consequences for the drivers’ well-being,
but can also cause the immediate impairments of driving performance.
For instance, anger is found to be associated with risky driving behaviour.
In light of this challenges, researchers are investigating the methods to monitor drivers’
psychological state. There are various existing works focusing
on detecting driver emotion using physiological signals or audio and video of drivers.
However, the fundamental barriers of these methods
is their privacy breaching property because video, audio or physiological signals are used.
Moreover, the accurate measurement of physiological signals is cumbersome.
In our study, we explored the possibility of driver emotion inference from non-intrusive
and privacy preserving modalities. we inspected two main goals in more detail.
First, we want to know To what extent can the emotions of drivers
be inferred based on (a) CAN data, (b) the traffic context, and (c) a combination of both (fusion)?
We performed the experiments in the wild. we modified 10 vehicles in order to access the
CAN data. In addition, two cameras were mounted on the dashboard to record the traffic context
and the driver’s facial expressions.
All drivers used the modified vehicles for daily drives.
In total, we obtained round 700 hours of valid data.
The GPS heat-map of all aggregated driving activities showed that
our data has a wide coverage around central Europe.
To record driver emotions while driving, we used the facial expression algorithm
as the annotation tool to obtain the emotion ground truth.
However, facial expressions are instantaneous,
as you can see in the figure. This means, when a driver is happy or angry,
he will only elicit corresponding facial expressions occasionally
Therefore, it is important to aggregate this high fluctuate detection into a stable representation.
To do this, we divided driving data into 5min segments
and computed the average level of each emotion. An emotion label of a segment is considered as
high class if its average presence is above personalized median value and vice versa.
Therefore, the objective of our final prediction was to determine whether a
driver’s emotion is positively or negatively deviated from their personal baseline.
We leveraged recurrent neural network and build 3 models:
namely CAN-Bus modal, which takes CAN bus as the only input data and similarly video model and fusion model.
We evaluated our models under two settings, namely the personalised model setting
and leave-one-subjct-out settings. Overall,
fusion model achieved the best performance with slight advantage over CAN model or video model
This indicated that driver emotions can be infered from either driving behaviour or traffic context.
Furthermore, it should be noted that leave-one-subject-out model does not perform
as good as personalised model. This is expected as emotion is very a personlised percepation
and each individual can perceive the same event with very different feelings.
Next, we compare our approach with physiological signal based method.
To this end, we replicated the state-of-the-art heart rate variability
based emotion recognition algorithm. The heart rate variabitlity of the
drivers is collected using firstbeat, which is a medical-grade device
Overall, in both personalised and leave-one-subject-out setting,
the vehicle data based method outperform heart rate variability based method.
The discovery is very encouraging as it showed that driver emotion recognition can be done
via non-intrusive and privacy preserving manner.
Our result revealed a great potential of non-intrusive driver emotion recognition.
Nevertheless, we need to address existing limitation that should be explored in the future.
First, our method relies on deep learning technique.
The black-box inference of driver emotions does not reveal transparent decision rules and
could erode the user’s trust in such system. Second, in leave-one-subject-out (LOSO) setting our model does not
perform as good as in the personalized setting. Generalisability remains a great challenge in
machine learning, we expect that more advanced techniques such as domain adaption or transfer
learning could be applied to solve this problem. Finally, Our current model is only capable
of performing binary classification of emotions. We envision that a more granular
recognition of emotion at different level could better benefit drivers.
Thank you for your attention and interest. Please
feel free to reach out if you have any further ideas and enjoy ubicomp 2021.