My name is tuochao chen, today I am very
honorable here to introduce our work
C-Face: Continuously Reconstructing
Facial Expressions by Deep Learning
Contours of the Face with Ear-mounted
Miniature Cameras
Facial expressions are a
very important nonverbal communication
channel we use every day to express ourselves
to one another.
They can be used to analyze people’s social
and emotional lives.
So, recognizing these expressions can
help computers to better interact with humans.
The most common way to recognize
the facial expression is just to place a camera
in front of the human.
However, this method may fail during the mobile situation
like walking or other daily activities.
To solve the issue, we can just wear the camera
on our body like this
. Are you willing to wear this in your
life? I think the answer is obvious
So, researchers have put forward different
kinds of wearable devices for
facial expression recognition.
However, most of them are still obtrusive
and could affect daily activities.
In addition, they can only classify discrete
expressions, none of them can continuously
reconstruct full expressions.
So, in this paper, we introduce C-Face
, a minimally obtrusive ear-mounted device
that can continuously reconstruct facial expressions.
Here is a quick demo for C-face,
the left image is captured by a front camera
and the right image shows our prediction results
using a face mesh model from Unity.
Now, let me introduce how C-face works.
We found that when people make different facial expressions,
the contours of their face change differently.
We will first introduce our ground-truth collection method and then our data pipeline.
For ground truth, we placed a camera in front of the user’s face and then used dlib library to extract facial landmarks as shown here.
For our data pipeline, we first preprocessed the raw images of the facial contours by skin color extraction and binarization.
Then these binary images were fed into a CNN framework that outputs the final facial landmarks.
In our user study, we conducted three experiments:
The continuous reconstruction study verified how C-Face performs for continuous reconstruction tasks.
The remounting study evaluated how C-Face performs after device remounting.
Lastly, the face mask study tested C-Face’s performance while wearing a mask.
I want to first introduce our evaluation metric before diving into the details of our study.
To evaluate error between prediction landmarks and ground-truth landmarks, we put forward a new evaluation metric that we call MMAE.
MMAE is the absolute error of only the major landmarks in the most active areas of the face, as shown in the video on the right.
In our user study, we asked 9 participants to continuously repeat the facial expressions displayed in the video.
Then we collected training and testing data separately, and used all of the frames to train and test our deep learning model.
Here are the results of our reconstruction study. On the left, is the average of the MMAE scores across all 9 participants for both earphones and headphones.
On the right, is a bar chart of the MMAE of each participant for both earphones and headphones.
Here is a visualization of our reconstruction results to better illustrate C-Face’s performance.
The middle image is the face captured by the front camera.
The left face mesh model is driven by landmarks from dlib library. The right one is driven by our predictions results.
In the remounting study, we used the data before remounting to train the model and used the data after remounting to test the model.
Here are our remounting results.
In the face mask study, participants wore a cut-out mask for ground-truth collection for the training data,
and a regular uncut mask for the testing data. The video on the right illustrates our reconstruction results.
Admittedly, C-Face still has room for improvement. For instance, our current model is user-dependant.
More people’s data needs to be collected for further study.
Also, complex lighting conditions and backgrounds affect segmentation, which could be improved with better software and hardware.
We could also work on lowering the computational burden by reducing data dimensions.
Finally, our current ground-truth dlib only reports the landmarks of the mouth, eyes and eyebrows.
Maybe in the future we can apply other higher-resolution methods, like iOS does with FaceID, to acquire more detail like wrinkles and dimples.
in conclusion C-Face is the first ear-mounted device that can continuously reconstruct full facial expressions. Thank you