My name is tuochao chen, today I am very honorable to introduce our work
C-Face: Continuously Reconstructing Facial Expressions
by Deep Learning Contours of the Face with Ear-mounted Miniature Cameras
Facial expressions are a very important nonverbal communication channel
we use extensively to express ourselves to one another
Because Facial movements have many degrees of freedom,
it allows the face to transmit high resolution information
So facial expressions can be used to analyze human social and emotional life
Also, recognizing these expressions will help computers
to better communicate with humans
Currently, the most common approach for computers to recognize facial expressions
is to place a camera in front of the face,
like the video call you need hold the smart phone
However, the front-facing camera method has some limitations in the real-world.
The first issue is blocking.
Recently, due to Covid19, it is essential for people to wear masks.
However, masks block the mouth.
Here is an example of a computer trying to recognize facial expressions,
while the user is wearing a mask.
The mask will really confuse the front camera,
and make it wonder where is the mouth?
Secondly, the front-facing camera method can be impractical in daily life.
Still take a video call as example, in a mobile setting,
it is inconvenient and sometimes unsafe to hold a camera in front of you
Also when doing daily activities that occupy the hands like cooking,
it is still difficult to position the camera to capture the face.
To solve these issues, one obvious solution is to
just wear the camera on yout body like this, or like this.
I just want to ask you a question
Are you willing to wear these when you are cooking or walking on the street?
The answer is obvious.
So we have three indexes to evaluate the facial reconstruction system
the first is reconstruction ability
if the system can continuously reconstruct facial movements
the second is mobility (how well the system can handle mobile settings)
the third one is to evaluate how obtrusive the device is
These are two methods we have just mentioned based on frontal cameras,
even if they have strong reconstruction ability,
they are obtrusive and cannot handle mobile settings.
So, some researchers put forward different kinds of wearable devices,
using acoustic signals, EMG or other different kinds of sensing technology
However, Even if they can work in mobile situations,
they are still kind of uncommon and obtrusive
So other researchers have put forward more common and unobtrusive devices
but they can only classify the discrete facial expressions, bot continuous recontrsuction
So, in this paper, we propose a novel system called C-Face
a minimally obtrusive device that enables mobility,
can continuously reconstruct the facial expression
Here is a quick demo for C-Face.
In the name: C-Face, the C stands for contour.
The main idea of C-Face is to use facial contours to continuously reconstruct facial movements
We use 42 facial landmarks from the dlib library
to represent the facial movement - including the mouth, eye and eyebrow
That is, in our work, we utilize these 42 landmarks as ground truth.
Our system is embedded into earphones and headphones
because headphones and earphones are widely used in our daily life
and minimally affect daily activity
The small rgb cameras are deployed on the earphones and headphones
to capture the contours of the face.
Since they have a high resolution, we believe it is possible
for cameras to provide enough information for continuous reconstruction.
Next, to verify our thoughts, we conducted a preliminary study
where we embedded two cameras into headphones (one on each)
We visualized the facial contour images captured by these two cameras in video.
We just found that when making different facial expressions,
the skin of the facial contour would deform differently.
Now let me introduce how we utilize the facial contour for continuous reconstruction.
Firstly, the facial contours captured by the ear-mounted cameras
would be firstly preprocessed with skin color extraction and binarization.
Next, the processed images of both the left and right side
are inputted into a convolutional neural network model.
Then the output of the neural network for both sides is inputted into a matching model
and finally outputs the 42 landmarks.
For ground-truth collection, in the study, participants is asked to sit in front of a rgb camera .
RGB camera would capture the front face and then we apply dlib library to generate ground-truth
To evaluate the error between the prediction landmarks and the ground-truth landmarks
we select two error indicators
Our first error indicator is the mean square distance (MSD) - a very common error indicator
You can see the right image, the red points are the ground truth and blue ones are our prediction results.
The MSD works by calculating the physical distance between each landmarks
of ground-truth and predictions and then calculating the average value as the MSD.
However, some landmarks on the face don’t move much when we make facial expressions.
Here is the video, the black curve is the trace of landmarks.
You can see that some points (such as eye canthus) do not move a lot during facial expression
So we put forward a new indicator called MMAE, which only calculates the error of the landmarks which move a lot.
So even if MMAE actually increases our error, but improves the quality and integrity of our results reporting.
During study, 9 participants are asked to continuously repeat
15 widely-used facial expressions for data collection as shown in video.
For the training dataset, the participant is asked to repeat the 15 facial expressions 5 times.
While they do this, we continuously collect all the frames to train the model.
Then, for the testing dataset, participants repeat them twice and all the frames are used to test the trained model.
Then we use the training and testing dataset to train a deep learning model and calculate the MMAE over 9 participants.
For earphones, the MMAE is 1.43mm and for headphones, the MMAE is 1.39mm.
To better illustrate the meaning of MMAE, we have this visual of participant 1 doing the expression |O:|
This is an example of landmarks where MMAE is at a minimum comparing prediction and ground-truth.
Next, we visualize the frame with maximum MMAE comparing prediction and ground-truth.
You can see how the difference between prediction and ground-truth becomes as MMAE increases.
Here is the video of our continuous reconstruction results.
The image in the middle is the face captured by the frontal camera
and the image on the sides are the facial contours captured by ear-mounted camera.
The white points on the face are the prediction landmarks.
To better present our result, we also utilize a 3D human head model from Unity to visualize our results.
The head model on the left is driven by dlib landmarks
and the head model on the right is driven by prediction landmarks.
Remounting is a challenge for wearable devices, since it causes the sensor shifting.
So, we also conducted a remounting study to test the robustness of C-face after remounting.
Specifically, we asked the user to remount the device and then collected new testing data after the remounting.
We then used the data after remounting to test the model trained by the data before remounting.
Since the camera shifting causes a transformation on the captured image
the data augmentation is applied to relieve this shifting effect.
Before input into the deep learning model, random transformation
would be applied on the binary image such as translation, rotation and scaling.
The result shows that data augmentation can help a lot to improve the performance of our remounting study
Also, we have found that headphones performed better than earphones in the remounting experiment,
And that may be because a larger freedom of movement when you wear earphones,
compared to headphones, which could cause greater camera shifting.
We also tested our system when the face is partially blocked wearing a mask.
In this study, some participants were asked to collect
training data and testing data wearing the cut out mask for ground truth collection.
The MMAE is similar compared with the results without a mask.
We also collected a new test dataset with the user wearing a regular uncut mask to simulate mask wearing in the real world.
Here is the video for this reconstruction result.
below face is just used to illustrate what of facial expression the user is doing when wearing the uncut mask
Admittedly, C-Face also has some issues which need to be improved.
Firstly, our current model is user-dependant, which means we need to collect new data for each new user.
In the future, we will collect data from more people and conduct further study on user-independent model.
secondly, like other computer vision systems, complex light conditions and backgrounds
can lead to poor image segmentation results and poor reconstruction performance.
So in the future we can improve the algorithm, for example, training a deep model for segmentation.
lso, we can improve the hardware, for example, by utilizing smaller depth cameras or IR cameras
Thirdly, for the earphones, if the user has long hair it may block the cameras and the system could fail.
Fourthly, we can also convert 2D images into a 1D curve and
deploy a more lightweight machine learning framework
to lower the computational burden and improve power efficiency.
Finally, we can also improve our ground truth collection methods.
Currently, Dlib only tells the landmarks of the mouth, eyes and eyebrows.
However, facial expressions also contain other information (like wrinkles and dimples),
maybe in the future we can acquire facial landmarks in more detail like FaceID in IOS.
In conclusion, C-Face is the first wearable sensing device using
minimally invasive common form factors that can continuously reconstruct
full facial expressions with the position and shapes of the mouth, eyes and eyebrows.