hello everyone
I'm Zisu Li from the
Hong Kong university of Science and Technology
today I want to present my work
enabling voice accompanying hand-to-face
gesture recognition with cross device sensing
this work was conducted in
the pervasive human computer interaction laboratory
at Tsinghua University
and was collaborated with HKUST APEX lab
So what is a Voice-Accompanying Hand-to-Face gesture
from linguistic research
people tend to perform
multi modalities like facial expressions
body movement and gesture with
speech simultaneously to express emotions
and these structures are mostly performed by hand
near the speaker's face
which is defined as hand to face
hand to face gestures are proven to be natural
expressive and more related to the speech
by the existing research
we could find
example VAHF gestures
of gestures in emojis like the picture shows
these gestures are all performed by the
hands near or on the face
and
people tend to perform these gestures in conversation
to express some emotions or narratives
the paper is about
exploring
and recognizing voice accompanying hand to face
gestures by cross device sensing
the main contribution of our work
was to conduct a comprehensive study for
the VAHF of gesture set
and then we proposed a sensor fusion technique
for recognizing it
lastly we present some use cases for VAHF of applications
our work was really
motivated by the falling observation and
prior research problem
voice input
has become a natural and always available interaction
modality for variable devices
such as earphones and smartwatches
however the modality control
for example the wake up state
the selection of the target device
invoice interaction is still a challenging problem
due to the implicitious
of modality information in speech
and restricted NLP techniques
in this case as figure one shows
people have to repeat the hot words like hey
Siri to wake up the
device or enable multiple runs of dialogues
we can look at the example from the prior work
yell and or use the simple hand on mouse gesture
to replace wake up words
in voice interaction
and this gesture is more accessible
than a physical or software button
especially when the device is not in the user's hands
so our motivation is to enable
a recognizing hand to face gestures to
remove the extra burdens of
modality control
and implemented it
for voice interaction on wearable devices
in our observation session for the firstly
we asked our participants to brainstorm VAHF of teachers
as many as possible
regardless of application scenarios or tasks
and we got 15 VAHF of teachers in this process
and now we conducted the subjective evaluation on
Seven Points like the scale question
now on usability social acceptance
disambiguity and fatigue
this is our result of subjective evaluation
the highlighted eight gestures will result after our
optimization process
to make our gesture set representative and easy to use
we optimized our subjective
result by the following principle
we categorized
gestures with three
morphology features which are the contact position
the contact type and the occlusion state
as these gestures could yield the same
sound propagation path
and potentially have similar semantics with
application scenarios
for example as the picture shows
this stretcher's contact position is the mouse and the
cheek its contact type is the palm
and the occlusion stayed in
the propagation path from the mouse to the year is
part partially included
this table shows the categorized
results of the three dimensions
we chose 1 gesture from each category
with the highest subjective scores
for our final gesture set
for example we found four gestures all belong to EFH
gestures which are similar in morphology and easy to
cause ambiguity when used
so we chose gesture one to our gesture set
as we uh optimized the initial results
we got eight VAHF of gestures with one
empty gesture as our finalized gestures set
after finalizing the gesture set
we thought to find out efficient hardware forms
and algorithms to recognize these gestures
so the 1st question we need to think about is
what types of features
would VAHF gestures yelled
here we are
summarized some of the key features that make VH
of gestures distinguishable
the main category of gestures we observed is that
hand to face gestures
would influence sound provocation to
different microphone channels
making differences in amplitude
phase and frequency response
taking this gesture
holding up the hand beside the mouth as an example
the sound propagation pass to to the right yabad
was blocked by the hoding up hand
leading to difference
in amplitude and frequency response
also the
microphone positions of a specific gesture brings
a unique phase difference
Another the category of
features is that the hand position
attitudes and movements
which can be potentially detected by variable devices
taking these considerations
the design goal of our recognition system is to sense
VAHF gestures with these distributed sensor notes
by observing
uh channel differences
which was
three types of commodity devices as censored nodes
a pair of ANC earbuds some
each earbuds has
one feed forward
microphone and one feedback microphone
a smart watch with one microphone and one speaker
a smart ring with 1 1 microphone and one IMU
in total there are 6 1 microphones one speaker
and one IMU in our sensing system
here is our hardware personal type
we used a zoom six ah
we used a zoom H six recorder as the audio card
to collect these six channels of audio input
to create
more precise features indicating face difference
we let the
speaker play ultrasonic chirp signals in loops
all the audio data and IMU data are streamed to a Python
backend synchronously
this figure shows an
overview of our algorithms consisting of three
sub models the vocal model
the ultrasound model and the IMU model
for more details of each channel
please refer to our paper
for the evaluation
we collected audio samples from users and performed
leave-one-out
cross validation on data set
we enhanced the data
by randomly cropping and shifting
the original audio samples
we conducted three of evaluation tasks uh
to factorial evaluation on device
combination and model selection
an evaluation on a reduced gesture set and an uh
ablation study
this table shows the results of the
two factorial evaluation
the levels are listed below the table
from the table we can figure out
that the future model achieved the highest accuracy of
91.5% for the
nine class classification
the accuracies are constantly improved by bringing
more device channels and using future models
on the right side we visualize the confusion matrix
of different models
the left
table shows the results on the reduced gesture set
which means our
sensing pipeline achieved a high usable accuracy for
signature VAHF gestures
the right table shows the abolition of pretraining
dropout and the
warm up schemes have specific levels of
negative effects on the performance
here is the demo
for example
if the user wants to know how to make egg fried rice
they can perform
different VAHF gestures to redirect their voice input
to different targets such as asking Siri
uh searching on Google and
sending a voice message to mum
the video shows the real time recognition results
how to make egg fried rice
thank you for your watching
please refer to our paper for more details