Hi everyone, My name is Songyun Tao, today I am very honorable here to present our work - NeckFace
continuously tracking full facial expressions on Neck-Mounted wearables together with Yaxuan and Tuochao
This work is conducted at Cornell University.
Human facial movements convey emotions, help us communicate nonverbally
and perform physical activities, such as eating and drinking.
Recognizing these expressions will better promote interpersonal interactions in computer-mediated scenarios.
For example, bringing facial expressions to VR can enable
the important emotional communication channels in the virtual meeting besides the voice communication.
In addition, facial expressions is a critical indicator of emotion and mind.
Enabling the devices to track one’s daily facial expression can better facilitate mental healthcare.
The most common and intuitive method to recognize the facial expression is just placing a camera in front of one’s face.
Yet, holding the phone in some situations like walking is inconvenient or even dangerous.
To deliver health benefit, one need to do long term inobtrusive tracking,
but It is also not practical to wear a camera in front of our face for daily monitoring.
Thus we have to find other solutions.
Some researchers put forward some wearable devices for facial expression recognition
using smart eyewear, acoustics signal, capacitive sensors, strain sensors, emg, and other different sensing technology.
However, most of them are still obtrusive and affect our daily activity.
In addition, they are all limited to classify the discrete expressions, not continuously reconstruct full expressions.
Our lab has developed the first ear-mounted face tracking expression wearable: C-Face.
Compared to C-face representation with 42 2D points,
NeckFace reconstruction achieves much better performance with even fine-grained cheek changes.
Furthermore, since infrared cameras capture only partial faces and filter our background,
NeckFace performs much better in terms of privacy.
We present NeckFace, the first neck-mounted wearable sensing technology
that can continuously track the full facial expressions using a neck-piece embedded with infrared (IR) camera.
Next, my co-author Yaxuan will introduce the design of our system.
The idea is to reconstruct the whole facial movements by partial information of chin, jaw and mouth
captured by an infrared camera wearing on the neck.
From our observation, different facial expressions or head rotations would lead to the difference illustrated in the IR images.
A customized deep learning pipeline based on Resnet 34 is developed to learn the captured IR images
to fully reconstruct facial movements and head rotation.
Data augmentation is applied on the training set to make the system more robust to motion noise.
For obtaining ground truth for our deep learning model,
we use TrueDepth camera and ARKit offered by Apple to extract full facial expressions.
Compared to the traditional CV library such as Dlib,
this method can extract high-quality and fine-grained facial features from 30,000 invisible dots from true depth camera.
It contains 52 blenshape parameters and three head rotation angles.
These blendshape parameters represent detailed movements of face with the range from 0 to 1000.
For obtaining IR images of our face,
we build IR camera with LED, camera, and filter on two common neck-mounted form factors: necklace and neckband.
Several iterations of the design is to find a balance between size, shape, and stability.
The necklace is lighter using only one camera.
The neckband uses two cameras covering a wider area, which helped achieve better performance.
For evaluating our system, we conducted a user study
with 13 participants and asked them to repeat 8 kinds of facial expression while rotating their head.
Then we collected the training data and testing data separately and use all the frames for our deep learning model.
In our user study, we conducted three experiments in total.
Sitting Study is to verify how NeckFace performs for the continuous facial expressions
and head rotation reconstruction task in a static situation.
Remounting Study is to evaluate how NeckFace performs after device remounting.
Lastly, walking Study is to test the robustness of NeckFace in the dynamic situation with device shaking.
Next, my colleauge Tuochao will introduce the results and future work of our study.
To evaluate the performance of our method, we calculate the Mean Absolute Error of 52 parameters
and three head angles between the output of the model and the output of ARKit.
To help better understand the meanings of MAE, we generate a list of facial expressions with different MAE.
When the MAE is lower than 40, the prediction result is visually highly similar to the ground truth.
We list MAE in the sitting study before and after remounting in the left tables.
In the right video, the middle image is the face captured by the front camera
and the left human face model is driven by ARkit from Iphone. And right one is driven by our predictions results.
There is no significant difference between the performance before remounting and after remounting by running ANOVA test.
Here is the MAE for our walking study, we also visualize our prediction results in the right video.
In addition, we have also have develop a real-time demo using the system pipeline as shown on the right.
The human head model on the back screen is our real-time prediction results created using Unity.
Like any camera-based system, sunlight’s inference is an issue for our system,
because the IR light in sunlight will affect the captured IR images as shown in the left images.
So, we have explored advanced hardware with better exposure strategy to get good performance under sunshine.
In the future, we will further reduce the device size using customized PCB board and smaller camera.
We will also decrease the LED duty cycle and low-power chip to lower device power consumption.
For more detailed information, please check our paper! Thank you!