Hello, my name is Laxmi.
I am a PhD student in Human-Computer Interaction lab at UC Merced.
In this talk, I will present LipType, a silent speech recognizer augmented with an independent
repair model.
Speech recognition is unreliable in noisy places, compromises privacy and security when
around strangers, and inaccessible to people with speech disorders.
A system that can understand speech by reading the lips of a speaker can mitigate many of
these challenges.
Existing lip reading systems either use external sensors or process lip movements to understand
what the user is speaking.
However, these systems are slow and error prone.
LipNet is a most widely used end-to-end model for phrase-level lip reading that maps a variable-length
sequence of video frames to text by making use of a deep three dimensional CNN, two bidirectional GRUs,
followed by a linear softmax layer and a CTC loss.
On top of that, it uses a beam search and five gram character-level language model for
decoding.
Although LipNet has proven to be promising, it has several limitations.
The use of 3D-CNN neglects the hidden information between channel correlations in spatial and
temporal directions, which limits the performance of its architecture.
Further, a deep 3D-CNN increases computational complexity and memory intensiveness.
We address these issues by combining a shallow 3D-CNN and a deep 2D-CNN integrated with squeeze
and excitation blocks to capture both spatial and temporal information.
This hybrid model captures the channel correlations to improve the quality of feature representations.
Moreover, it is computationally lightweight. We call this optimized model LipType.
We trained and tested LipType model on the GRID dataset, the same dataset the parent
model was trained on.
For evaluation, we compared LipType with the parent LipNet model.
Results revealed that LipType outperformed LipNet.
It showed 47% reduction in error rate, 39% increase in words per minute, and 8.6 seconds
reduction in computation time.
In further investigation, we identified two common problems with image-based lip reading model.
First, they do not perform well under poor lighting conditions and second, they make
various lexical and linguistic errors.
Thus, we developed an independent repair model that compensates for poor lighting conditions
and corrects potential errors in output using a custom language model.
For light enhancement, we augmented a lowlight image enhancement network called GLADNet by
using the most appropriate loss function and the layer optimization through multiple experiments.
It takes a sequence of low-light images and outputs enhanced images.
These enhanced images are compared with their corresponding normal-light images to compute
the loss, which is then back-propagated to fine-tune and optimize the model.
The error correction module predicts and automatically corrects potential recognition errors made
by the recognizer.
It comprises of two sub-modules: an error minimization module that corrects potential
errors in the recognized character sequence using deep denoising autoencoder and a sequence
decoder module that converts corrected character sequence to meaningful word sequences using
spell-checker and a custom language model.
Both sub-modules were evaluated and refined based on empirical studies.
To demonstrate the repair models' independence and effectiveness, we compared three popular
pre-trained speech and three popular pre-trained silent speech recognizers with and without
the repair model.
The study involve twelve participants and collected diverse data in real-world conditions.
For speech, we collected data in indoor, outdoor, and quiet noisy conditions.
For silent speech, we collected data in dark, dusky, and day lighting conditions.
Results showed substantial improvement in the performance of all speech and silent speech
recognition models.
On average, speech models showed 32% reduction in error rate, 8.1% reduction in words per minute,
and 5.8 seconds increase in computation time, whereas, silent speech models showed
57.2% reduction in error rate, 10.3% reduction in words per minute, and 7.9 seconds increase
in the computation time.
To summarize, we develop LipType, an optimized LipNet model for improved speed and accuracy.
We then develop an independent repair model that can account for poor lighting conditions
and potential recognition errors in various speech and silent speech recognition systems.
The findings confirm that the model can be used independently with various recognizers
for substantially improved accuracy.
Thank you