Hi Everyone! I am Dhruva Bansal and I will be presenting our paper titled “CopyCat:
Using Sign Language Recognition to Help Deaf Children Acquire Language Skills”
Over 95% of deaf children are born to hearing parents, but most hearing parents
do not learn American sign language, or ASL, sufficiently well to teach ASL to their children.
These children, in turn, often lack
basic communication skills and the short term memory associated with language communication,
leading to social isolation and language deprivation syndrome. Language Deprivation
Syndrome is associated with a significant increase in abuse, mental health issues, and suicide.
CopyCat, an educational game, seeks to help deaf children acquire language skills
required to avoid language deprivation syndrome. The game features a character to which the player
must sign three to five sign phrases to proceed. The game uses sign language
recognizers as a key component of the game to help children acquire generative language skills.
CopyCat uses the Game UI that you can see here, with a hint video performed by the instructor
and a mirror of the signer. The scene that the user must describe is presented on the right.
Here is a video of the complete game in operation. The scene presented corresponds to lion above grey
wall and the user signs correctly. The recognition system verifies this signing and proceeds.
Here is the game operation in the case of incorrect signing.
The scene corresponds to snake in grey wagon but the user signs lion above
wall. The recognition algorithm realizes that and blocks the user from proceeding.
A Wizard Of Oz experiment conducted using the previous version of CopyCat revealed
that students who played CopyCat doubled their scores on various measures of language skills.
The game’s effectiveness naturally depends upon the accuracy with which it classifies
sentences as correct and incorrect. Previous versions have relied on collecting data using
expensive custom hardware, equipped with accelerometers along with Hidden Markov
Models (HMMs) for performing this classification. However, the need for such specialized equipment
restricts the game’s scale and impact due to its cost and hardware accessibility.
We improve upon these previous versions by incorporating off-the-shelf cameras
like Azure Kinect for data collection, thus improving the game’s ubiquity and scalability.
We collected 4K RGBD from 12 adults at
Georgia Tech. For our phrases, we used a noun-preposition-noun
structure with optional adjectives, such as “alligator above orange wagon”.
We extracted hand, face, and body pose features with Google MediaPipe,
Microsoft Kinect, and AlphaPose. While MediaPipe and Alphapose give only 2-dimensional features,
Kinect provides us with 3-dimensional features on the user. Our features can also be found online,
as we have open sourced all of our data while keeping users anonymous.
We used extracted feature data as input to our recognition models.
We hypothesized that hidden markov models are best suited for ASL recognition
due to their great performance on time series and pattern recognition problems along with
low training data requirements when compared to deep learning models such as Transformers.
We evaluate our recognition systems on all 3 pose estimators using HMMs and Transformers
in 3 experimental settings - User Adaptive, User Dependent, and User Independent. Surprisingly,
we find that 2D pose estimators - MediaPipe and AlphaPose - achieve parity with Kinect.
We found that HMMs consistently outperform Transformers in all 3 experimental settings,
regardless of the pose estimator being used. Poor performance in Dependent and Independent
settings suggests that Transformers don’t generalize well over the current dataset.
Our codebase along with the dataset used for this analysis is available via our GitHub repository.
Building upon the success of a recognition system, we next hope to build a verification
system for performing the same task. Zahoor et al. demonstrated that their verification
algorithm outperforms recognition algorithm on 3 multimodal ASL datasets. The verification system
improves performance by leveraging information about the ground truth scenes themself.
Finally, we will also conduct a large-scale user study on deaf children to verify the educational
benefits of the new CopyCat game. Thank you for watching our presentation! If you have
questions or would like to learn more about our project, feel free to contact us.