Hi everyone. In this talk, I will introduce you to our recent work about using acoustic
sensing methods to support upper facial action recognition for smart eyewear.
Let me first discuss the motivation of our design. Smart eyewear (e.g., AR glasses) is considered
to be the next big breakthrough for wearable technologies because it put information right
in front of the eyes which creates an immersive human-computer interaction experience. However,
the interaction scheme of state-of-the-art smart eyewear mostly relies on the touchpad. Unlike
smartphones and smartwatches where touch-based interaction requires only finger tapping,
to interact with a glass-mounted touchpad, a user needs to raise his/her arm to reach the touchpad.
Thus, frequently interacting with smart eyewear may cause fatigue and is not friendly to the
elderly and disabled users. Although most smart eyewear is equipped with a voice assistant,
there are situations when voice commands are restricted, such as during a meeting or in
a library. Therefore, a new hands-free and quiet interaction system for smart eyewear is desirable.
So the question is: can we design such a system for smart eyewear?
If yes, then what is the interaction vocabulary, and what is the input media?
Recent works have shown the possibility of using upper facial action or UFA as
a hands-free input modality and we believe such method is preferable for eyewear because
eyewear is, by nature, close to our face thus it is easy for an eyeglass to sense UFAs.
As for the sensing modality, we believe the acoustic signal is, among others, more
suitable for this task because it is accurate, contactless, and most importantly cost-efficient.
Therefore, our solution is to use acoustic sensing method to sense
the following five UFAs, cheek-raise, brow-raise, brow-lower, wink, and blink.
This is our prototype, we customized the Seeed 4-mic linear array to a circular array to fit into
an eyeglass frame. The 4-mic array together with a pair of earphones, which serves as speakers, are
connected to the Seeed voice accessory board and the entire system is powered by a Raspberry Pi.
The main idea of our software design is to use channel state information or CSI as a tool to
sense the skin deformation caused by UFAs and use deep learning techniques for classification.
The software system consists of three components:
OFDM-based CSI estimation, CSI signal processing, and UFA recognition. In the rest of this talk,
I will briefly introduce the three components. For the design details, please refer to our paper.
In the OFDM-based CSI Estimation module, we let the speakers transmit an inaudible OFDM signal
and we extract CSI from the received signals by computing the channel frequency response.
An example of the CSI signal of an individual performing a brow-raise is shown here.
Note that our OFDM scheme has 16 operational channels and the signals are plotted as
time sequences in the I-Q domain. After extracting the 16-channel CSI signals,
we use the PCA algorithm to fuse the 16 channels into one single channel.
Next, we conduct feature engineering to extract features from the CSI signal.
We use short-time Fourier transform to extract the time-frequency domain features.
As shown here, different UFAs have different time-frequency features and the features
show a certain level of consistency across subjects. Except for the time-frequency domain,
since our eyewear has 4 mics and 2 speakers, which sum up to 8 links, different UFAs show
distinct characteristics in the link domain as well. For example, a cheek-raise shows high energy
at the last two links while a brow-raise shows high energy in the first two links.
Finally, after the 8-link time-frequency features are ready, we use a 3-layer CNN
for high-level representation extraction and classification.
Note that we use three techniques to prevent the model from overfitting. Among these three, the
confidence control constraint punishes the model when the model gives overlarge confidence on a
training sample. It is worth noting that although the features show some level of consistency
across subjects, the human factors still cannot be neglected considering the limited size
of the training data. Therefore, we use transfer learning to personalize the model to new users.
Specifically, in the personalization process, we freeze the model except for the last layer
and we retrain the last layer with only a few samples for each UFA from the new user.
We collect data from 26 subjects and our leave-one-subject-out validation shows
that our eyewear can achieve satisfactory accuracy among the target facial actions
and across different subjects.
Thank you for your listening and now I’m willing to take any questions