Hello everyone! I'm Dhruv and I'm a PhD student at the University of Toronto.
Today, I'll be joined by my co-author Sejal and we are very excited to present our work ExpressEar
which utilizes ear-mounted IMU sensors for fine-grained facial expression recognition.
This work was done with our collaborators Dhruv Sahnan, Jainendra Shukla and Aman Parnami at the Indraprastha Institute of Information Technology Delhi.
Facial expressions occupy a central role in human social interaction.
They are natural, intuitive, diverse and contextual which makes them instrumental in comprehending human social and emotional lives.
Currently, the most common approach relies primarily on camera-based systems.
Despite the high accuracy of these systems, they are limited by their obtrusiveness limited tolerance to occlusion and privacy concerns posed by camera-based interfaces.
In our work, we take advantage of the growing popularity and minimally obtrusive form factor
of the true wireless earbuds as ubiquitous sensing modality.
In recent years, numerous researches have devised unobtrusive ways of sensing facial expressions through mounting sensors on the ear.
however, these approaches usually fail due to limited or arbitrary selection of facial expressions.
or use of custom sensors which limit its practical adoption.
Addressing some of these limitations, we propose ExpressEar which is an ear-mounted IMU sensing technique,
capable of recognizing fine-grained facial expressions.
ExpressEar is based on the popular Facial Action Coding System (FACS) which encodes every arbitrary facial expression into its constituent Action Units (AUs).
Our system works on the basic principle that facial muscle movements cause deformations in the
ear canal which produces perceptible differences in the signal patterns of the IMU signals.
For the purpose of this work, we operate on the Nokia eSense platform.
We make use of its 6-DOF IMU sensor and stream data from both left and right earbuds at a sampling rate of 50 hertz.
The IMU data stream from both the earbuds is transformed into two-second long segments via a sliding window approach.
These segments are then passed into a bandpass filter and finally stacked together before feeding in into the neural net.
For facial AU classification, we make use of Temporal Convolutional Networks (TCNs).
We evaluated ExpressEar to investigate several important questions about facial AU sensing.
For the same, we conducted a study with 12 participants including 8 females and 4 males in the age group of 19 to 23.
We conducted the study remotely and the apparatus was the same as shown in this illustration.
The study followed a within-subject experimental design involving seven rounds of data collection from each participant as shown.
This resulted in a data set consisting of 24000 facial expression segments which were labelled by the participants themselves each time they recorded an AU in the android app.
We also included time-shifted windows of these labelled segments for augmenting our data set resulting in a total of 120,000 labelled facial expression segments.
We applied various learning algorithms in the collected data following a five-fold cross-validation scheme.
We found that the temporal convolutional network outperformed the rest with an average accuracy of 90.2 percent
therefore, we chose TCN for implementing ExpressEar and for all further testing
Next, we investigated the performance of our model across users. The main leave-one-user-out accuracy was 42.1 percent which although impractical,
is still well above chance for a 32 class classification. On the other hand per user classifiers yielded an average
accuracy of 89.9 percent across 32 AUs.
Finally, to assess the effect of the plurality of sensors we train per user models using data from only one IMU at a time.
The average per-user accuracy across all subjects was 79.3 and 79.6 percent for right and left IMUs respectively.
Although using data from both IMUs produced a significantly better result as verified by Friedman's test,
a single IMU augmented earbud can also be reliably leveraged to perform facial action recognition with decent accuracy.
The results so far depict the performance of our system in a controlled lab environment,
however, in a real-world scenario ExpressEar is likely to encounter unknown face related movements
like speaking and eating, and other motion artefacts arising from the whole-body motion of the user.
Thus we conducted two additional experiments to estimate the performance of our system
under challenging conditions of noise and user mobility.
First, we assess the distinguishability of the complete set of facial AUs from the absence of any facial expression (which we call baseline)
and other face-related movements like eating and speaking.
On training per user binary classifiers, we obtained an average accuracy of 99.2 percent.
The almost perfect result is unsurprising since there is a considerable difference in the SNR of the two sets of classes.
Next, we evaluated the effect of whole-body motion including walking and travelling in a moving metro on the performance of ExpressEar.
We determined the optimal set of potentially classifiable action units for both motion scenarios in accordance with their signal to noise ratios.
We use these sets of AUs for final classification and obtained an average accuracy of 84.34 percent for walking and 83.85 percent for metro.
In this way, we demonstrated the potential of ear mounted IMU sensors to detect
and recognize fine-grained facial expressions in multiple scenarios of daily living.
however, there are certain limitations of our work such as individual variability
that arises due to the heterogeneous nature of facial muscles.
Susceptibility to motion artefacts that leads to non-additive noise and finally acoustic sensitivity which remains something to be studied.
In conclusion, while technical challenges remain the main goal of this work is to
shift the community's attention to a privacy preserving unobtrusive and a ubiquitous
sensing technique that holds the potential to continuously monitor rich non-verbal cues
which encode facial expressions at the atomic level.
Thank you so much for listening to us!
Check out our paper for more details, and if you have any queries please feel free to reach out to us.