Hello, everyone. My name is Vimal Kakaraparthi.
I'm currently a PhD student in CS Department at the University of Colorado.
co-advised by Dr.VP Nguyen of University of Texas Arlington.
I'm here to present our system FaceSense
an ear worn device that senses a face touch event and classifies whether
the touch landed in a mucosal zone or a non mucosal zone.
This is a joint work between University
of Colorado Boulder, University of Texas Arlington,
Dartmouth College and University of Oxford.
The habit of face touch spreads pathogens
associated with COVID-19, Ebola, influenza, etc.
This involuntary habit is also associated
with conditions such as chronic eye rubbing compulsive skin picking and several
such obsessive behaviors which have a lasting effect on a person's body.
Existing devices assume hand-to-face
proximity or specific hand motion patterns to be face touch.
The innate problem with this approach is
that there are numerous daily activities such as eating, drinking, adjusting
glasses, answering a call, etc, that would be identified as a face touch.
This leads to user mistrust of the model and an ineffective habit reversal
ear-
worn thermal cameras capture an approaching hand and spatial hints
in a privacy preserving manner, and physiological sensors capture
skin deformations caused by face touch in the form of skin impedance changes.
This information can be fused to identify
a face touch event and classify whether the touch landed in a sensitive zone,
such as eyes, ears, nose, or a non-sensitive zone of the face.
We propose FaceSense and ear-worn device that uses data from thermal images
and physiological signals to detect an actual touch and classify whether
the touch landed in a sensitive zone or a non-sensitive zone.
The first challenge in realizing such
a system is that basic facial activities such as talking, drinking, eating, etc
cause skin impedance variations similar to face touch.
Second challenge is that the existence
of thermal objects in a background can ruin a thermal profile of a hand.
The third challenge is to design
an algorithm that effectively fuses heterogeneous data from physiological
subsystem and thermal subsystem to learn proper representations of a face touch event.
Our device is designed to overcome all these challenges.
Face touch system is composed of three major
components, namely the physiological subsystem, the thermal subsystem
and a Multimodal Neural Network, which we call TouchNet.
Physiological Subsystem uses a bio amplifier at its center.
Two channels of this bio amplifier are connected to a small area
near the ear, one on each side of the face.
These surface electrode connections are designed to capture skin impedance
fluctuations wherever the touch is happening on the face.
irrespective of the intensity of the touch.
Prior research found that facial activities
tend to generate equal skin impedance changes and EMG on both sides of the face.
However, face touch generates unequal skin
impedance fluctuations on both sides of the face and there is no facial EMG.
This can be observed in figures showing
cross correlation scores of two channels while doing the activities by themselves
and when the activity is compounded with face.
touch
thermal subsystem identifies the presence of a hand, which not only provides spatial
hints for touch zone identification, but also helps to identify face touch
events when they occur along with asymmetrical facial activities.
Thermal Subsystem consists of two ear mounted infrared thermal cameras that are
low power and have a low resolution of 8 by 8 pixels.
The subsystem has a field of your 120 degrees vertically and horizontally.
We implement background subtraction,
followed by the binarization of thermal images to single out hand
signatures from noisy backgrounds before feeding them to Touch Net.
Next, we design Touch Net,
a multimodal deep learning model that is composed of three main modules,
the feature extractor, which learns joint representation
from heterogeneous data, the user discriminator that improves cross
user performance by suppressing user dependent information,
and the face touch detector that actually classifies the data.
The implementation details of Touch Net are elaborated in our paper.
FaceSense wearable, consists of two flexible
earpieces manufactured using thermoplastic urethane.
These ear pieces ensure contact quality of surface electrodes,
positioning of thermal camera, social acceptability, and user comfort.
The average cross user accuracy of FaceSense with 14 participants is 84% with binary
classification. Touch zone identification currently requires
a small training overhead of five minutes for each participant to achieve a high
macro F-1 score of 90.8% and a micro accuracy of 90.1% with three classes,
namely no Touch, nonsensitive touch sensitive touch.
To learn more details about data analysis,
evaluation, prototype implementation, future work, etc.
Be sure to check out our paper at UbiComp 2021.
21. Thank you very much.