Hi everyone, My name is Giuliana and this work was done with my advisor Misha Sra at UCSB.
In my talk today, I will introduce the project, walk through the system design and the implementation and then go over our user evaluation and finally highlight some limitations and future work.
Imagine you would wake up one morning and out of nowhere, you cannot move one side of your face anymore. What would you do? Probably you would try everything possible to get back to normal.
Facial paralysis is the most common facial nerve disorder. It causes functional and aesthetic deficits that often lead to emotional distress and affect psychosocial well-being.
One form of treatment is mirror therapy, in which a mirror is used to reflect the healthy onto the unhealthy side of the face. The result: Patients see their healthy, fully reconstructed face while performing facial exercises.
While showing high potential, Mirror Therapy has several mirror-related drawbacks that limit its effectiveness, including: (a) reduced user motivation (b) lack of meaningful feedback and muscle education (d) mirror exposure anxiety and (e) inability to track progress over time.
Introducing FaraPy, the first mobile augmented reality mirror therapy system for facial paralysis. FaraPy provides real-time feedback and tracks patient progress over time using a novel deep learning model in the back end.
Our key contributions are as follows: We developed •The first AR-based real-time physical therapy feedback system for individuals with FP. •A novel lightweight deep learning model that runs on smartphones in real-time and outperforms state-of-the-art models. We enabled •The prediction of unilateral facial action unit intensities (AUI) independently for each side of the face, as opposed to traditional bilateral predictions of benchmark models. •A new FP dataset with unilateral action unit intensity labels. •An end-to-end pipeline with a fully working system that can work on mobile and other devices with front facing cameras
Our proposed Light Weight Facial Activation Unit (LW-FAU) model detects facial muscle activations and intensities based on detected facial muscle activations from the user’s camera input. FaraPy allows individuals to receive feedback on exercise performance at home, enabling them to continue their recovery outside of therapy sessions, which is an important contributor to successful recovery.
The system works as follows: (a) Camera data is processed in real-time by the proposed (b) Light Weight Facial Activation Unit Model, called LW-FAU. LW-FAU is an efficient convolutional neural network that detects muscle activations and intensities in real-time as users perform facial exercises in front of a mobile device camera.
(c) Left- and right-side facial muscle activation predictions are processed in (d) JavaScript, where (2) the paralyzed side of the face is identified as the one with overall lower activations. (3) Ratios per AU are calculated to provide (4) feedback on the symmetry of activation.
(5) The muscle education and mirror therapy filter is activated on the identified paralyzed side, and (6) facial expressions are scored for symmetry. (e) Activations and the facial filter are visualized in AR.
Our model outperforms the state-of-the-art model on benchmark data for detecting facial action unit intensity. Please refer to our paper for more details.
Our user study (n=20) shows high patient satisfaction and greater preference for our interactive system over traditional mirror therapy.
Finally, we would like to discuss limitations and research opportunities of our work: •1. Lack of FP Benchmark Models and Data Sets: There are no benchmark models or datasets for one-sided AUI values. Manual labeling of datasets is time, cost and in this particular case, expertise intensive. Therefore, we used the current state-of-the-art FAU-Net model for labeling our FIFA user dataset and the DISFA benchmark dataset. Future work could explicitly address the lack of accessible datasets on individuals with FP.
•2. Lack of Local User Data: We did not have access to participant device data. Due to COVID19, we were forced to conduct our study with remote participants. Therefore, we could not test our system on a single device under controlled but randomized conditions in our laboratory. Follow-up studies are needed to test the model’s performance on a single device for better comparison and standardization.
•3. User Study Sample Size: Although our sample size of 𝑠𝑎𝑚𝑝𝑙𝑒𝑠𝑖𝑧𝑒 = 20 was equal to or higher than that in benchmark FP user studies [8, 32, 33], the small number does impose limitations on interpreting the statistical results and generalizability. Future studies could be conducted to validate identified trends with higher statistical power.
•4. AU Selection: FP with synkinesis requires focus on facial exercises related to eye movements. This type of FP and related exercises are not fully supported by our current system which lacks explicit eye-tracking. For our model LW-FAU, we prioritized AUs related to the facial mouth parts and those were the main focus of FP exercises performed during the user study. Future versions of our prototype would be enhanced by incorporating eye-tracking to support synkinesis-related exercises.
Imagine a world where FP is not a threat anymore; Due to efficient treatments, accessible anyplace/time