Hello everyone this is Panneer Selvam Santhalingam from George Mason university
today i am going to talk to you about our work on environment independent ASL
gesture recognition using 60 GHz millimeter wave
signals we proposed a system called mmASL. This
work was done in collaboration with Alamin,
Ding, Parth, Huzefa from George Mason university
and Raja from Gallaudet university. Let's talk about home assistants.
Home assistants are everywhere they provide us a rich user interface to
interact with our everyday devices and make life simple.
Most of the current generation home assistant devices are predominantly voice based.
This restrict their access to Deaf and Hard of Hearing community
Close to half million people in the DHH community use American sign language
a visual mode of communication for the everyday interaction
American sign language involves hand gestures, finger-spelling,
bodily gestures and facial movements. Expanding the capability of the current
generation home assistant devices to detect ASL gestures will help us to
bring the home assistant devices to the DHH
community and reap the benefits that we are gaining.
Most of the current generation home assistant devices operate in two modes
the first is the wake word mode wherein the user initiates an interaction with
the device which is followed by the command mode
wherein the user actually interacts with the device.
In order to build a DHH home assistant device we need to solve several
challenges. The first one is the ability to detect a
visual wake word. In contrast to existing home assistant
devices, DHH hom assistant devices should be able to detect wake words in a visual
mode. The next one is the ability to detect
ASL gestures in several practical scenarios.
This involves varying lighting conditions the presence of an
interfering user as well as varying environment conditions.
In this work we propose mmASL a 60 GHz millimeter wave based DHH home assistant
system. 60 GhHz signals have been already adopted
for the next generation multi-giga bit wireless systems. Because of the higher
bandwidth and frequency 60 GHz signals have also been adopted for
several fine-grained gesture recognition systems. The system is built using Ni
+Si beam multi-fpga software radio platform.
Because of the directionality the 60 GHz systems
in contrast to other RF sensing systems like CSI are non-intrusive
is tolerant to the presence of other people as well as environment
independent. We propose a custom signal processing
pipeline which exploits the observed doppler spread in
recognizing ASL signs using doppler spread spectrogram.
mmASL operates in two phases the first phase is the wake word recognition phase
in this phase mmASL detects the wake word
and infers the users internet to interact with mmASL
as well as the actual spatial location of the user.
mmASL utilizes beam steering to scan the environment and generates a spatial
spectrogram. A spatial spectrogram is the
concatenation of the different doppler spread spectrograms generated in each
individual beam sector. The spatial spectrogram is further
passed down the line to a deep learning algorithm
which answers two questions whether a wake word was present or not
and the actual user location which is expressed as a beam sector
index
With the wake word detection, mmASL fixes the beam sector and moves to the next
phase which is the ASL gesture recognition.
In this phase the user performs different ASL gestures
the raw data which is recovered from the ASL gestures is processed using the
signal processing pipeline to generate the Doppler spread spectrogram.
The Doppler spread spectrogram is further processed downline using a
multitask deep learning model which utilizes ASL domain-specific
knowledge in classifying the ASL sign. We utilize the phonological properties
observed in the ASL language and encode them as auxiliary task
We utilize two auxiliary tasks one is whether there is a repetitive motion
present in the ASL sign or not. The second one is what is the moment
type whether it is perpendicular to the human body or parallel to the human body.
These auxiliary tasks complement the main task and helps in it better
performing the actual ASL gesture recognition.
We evaluate the performance of mmASL using separate data sets for wake word
recognition and ASL gesture recognition. The data set
includes close to 15 000 samples collected for 50
signs from close to 15 users. The data set was collected
from three different environments and includes several practical scenarios
involving an interfering user. In wake word detection we observed that the
CNN based deep learning model outperforms most of the traditional machine learning
models. We note that the CNN based algorithm can also
perform well in several different environments wherein we have tested for
the wake word. We note that because of the difference
in speed with which different users does the wake word
there is a small drop in performance in the wake word recognition for some users.
We evaluate the performance of mmASL in gesture recognition by comparing it with
several existing state-of-the-art systems like kinect and camera-based
solutions. The first thing is we observe that
mmASL can offer comparable performance with existing
state-of-the-art systems. We also note that the multitask deep
learning architecture helps mmASL to perform better in
different unseen scenarios involving an
interfering user, also when it was evaluated on new untrained users.
I would like to thank you all for listening to my talk.
I will take any questions from here thank you!