Hi! I'm Vinitha Gadiraju. I am a PhD student at the University of Colorado Boulder and
today I will be presenting our CHI paper, BrailleBlocks: Computational Braille Toys
for Collaborative Learning, on behalf of myself and my co-authors Annika Muehlbradt and Shaun
Kane.
So experts have found that Braille literacy is declining among blind students in the US.
In 1960, over fifty percent of blind students were identified as Braille readers. However,
in 2017, only about eight percent of blind students in the US were identified as Braille
readers. A possible catalyst for this decline includes rising accessible technology that
present other methods of reading like audiobooks and text to speech technology.
However, even with alternative reading methods available, education experts have found that
Braille literacy improves reading comprehension skills and increases chances of employment
for visually impaired people. However, learning Braille can be tedious and difficult, and
an isolating process for students. Our goal is to build tools that encourage visually
impaired people and sighted people to learn Braille together in fun and engaging ways.
Games can motivate children to interact with Braille and build fun learning experiences.
Researchers have created Braille games and mobile applications to teach basic Braille
concepts. BraillePlay, gBraille, and mBraille use haptic feedback on a phone's touchscreen
to represent a Braille cell, with the dots in the cell that make a Braille letter. These
applications include audio-based games like Hangman, to practice spelling and letter recall.
In the image on screen, there is a mobile Braille cell representation from the BraillePlay
interface. Users can double tap one of the six dots on screen to raise that dot in the
Braille cell. Users can use a two-finger swipe to submit the letter.
There is a small selection of Braille toys for visually impaired children to play with,
many of which are block-based. The top image is BrailleBricks, a commercial product that
Lego is releasing this year. BrailleBricks are Legos with slightly enlarged Braille on
the surface. Students can place legos together on a mat to build words and sentences. In
the image on the bottom right, we can see a similar approach from the company Tack-Tiles.
Tack-Tiles uses Lego-sized blocks with Braille on the surface. There are also larger blocks
with both print and Braille, as seen in the image to the bottom left of PlanToys Braille
A to Z Alphabet Set.
Blocks and tangibles have also been used outside of Braille education for visually impaired
children. In particular, interactive tangibles have been used to teach the basics of programming.
The image on the left shows connectible pods and the interactive programming application
from Microsoft's Code Jumper. Code Jumper uses a set of connectible pods to enable children
to construct programs that represent music and other audio. The image on the right shows
StoryBlocks, a set of code blocks that students can assemble to program fun, audio stories
as output. Each of these systems combines tactile interaction with audio output, to
create an engaging and accessible learning experience.
BrailleBlocks draws inspiration from current Braille toys, game applications, and tangible
computing and applies the overall approach of interactive, tangible blocks to Braille
education. BrailleBlocks is a smart toy block set that supports interactive games by tracking
the blocks and providing audio feedback for everyone and visual feedback for parents.
The top image on screen shows the three components of BrailleBlocks. The first component of BrailleBlocks
is the tangibles, which includes the blocks, pegs, and a frame that contains the blocks.
The second, is the associated web application, and the third is an overhead webcam to track
the blocks. The bottom image on screen is a close-up of the BrailleBlocks tangible set.
A Braille cell contains six dots that are numbered top to bottom, left to right. This
is shown in the BrailleBlocks in the image on the right. A Braille letter is determined
by which dots are raised, in this case which holes have pegs in them. We chose to make
enlarged blocks, rather than standard size, to support children still working on early
motor skills, so that they can easily grab the blocks and physically construct letters
using the pegs. The image on the left shows the dimensions in inches for a single BrailleBlock.
The block is 1.25 inches high, 2 inches wide, and 3 inches long.
Each block in the tangible set represents a single Braille cell. Children can place
pegs in the block's holes to build Braille letters. They can then place blocks together
to create words. In the video we just saw, we saw a user placing pegs in dots one, three,
and five of the BrailleBlock, to create the letter "O".
Each game in the system has a Check Word feature. This connects to a computer vision program
we wrote to help parents and other players who are not familiar with Braille determine
what letters the child has built. When parents press the Check Word button shown at the top
of the screen, the overhead webcam takes a picture of what the child has built. The picture
of the blocks and pegs inside the frame taken by the webcam is shown at the bottom of the
screen. In the computer vision program, first we clip the image using four colored markers
on the edge of the frame. This is shown in the image at the top of the screen. Then we
apply a mask to extract all the red from the image. This is shown in the image in the middle
of the screen. From there, we can determine what letter the block represents and the system
outputs the letters on a dialog box on the interface for parents to check what the child
has built. The dialog box is shown at the bottom of the screen, outputting the word
"ELEPHANT".
The system includes three interactive games. The first is the Animal Name Game, which uses
audio clues to practice spelling. The image on the left shows the animal selection chart
from the Animal Name Game, with images of a dog, elephant, duck, and sheep. The second
is Hangman, which uses audio for correct and incorrect letter guesses. The main screen
from this game is shown in the image in the top right, which has a text input form and
a set of interactive buttons. The last game is a Word Scramble that works on tactile letter
identification. Level 1 of Word Scramble is shown in the bottom right image on screen.
The image has Braille representations of the letters "t", "c", and "a", along with some
words the letters can unscramble to make, like "act" and "cat."
So this is a video walkthrough of the Animal Name Game. Parents select elephant from the
animal selection chart described earlier, and then play the animal sound. *Elephant
trunk sound*. That was an elephant noise. Now the child is building the word elephant
with the blocks and pegs. And then parents can check what the child has built using the
Check Word feature. And so on screen you can see that the dialog box says "ELEPHANT."
So in Hangman, parents can enter a goal word and then play a ding for a correct guess,
*ding noise*, and a buzzer sound for an incorrect guess, *buzzer sound*. So the word that was
entered was pickles, and the parents can use the Check Word feature to check what their
child has built in this game as well.
So there are five levels in Word Scramble, and they each increase in difficulty. So in
this video, the parent selects level 1, and there is a visual Braille representation of
the letters "t", "c", and "a", along with the words they unscramble to make, "act",
"cat", and "at". Parents can also use the Check Word feature here to see if their child
has unscrambled the letters correctly.
So we conducted a usability study to observe how sighted parents and visually impaired
children collaborated and learned with BrailleBlocks and to collect feedback about our existing
prototypes. The image on screen shows one of our participant families using the system.
On the left are two visually impaired children and on the right is their sighted parent.
The study consisted of individual sessions with five participant families. Four of these
families had one sighted parent and one visually impaired child. One family had one sighted
parent and two visually impaired siblings. There are cartoon drawings on screen of a
parent a child, each representing our participant families.
We asked participants to explore the games I outlined earlier and encouraged them to
use the system however they felt best addressed their child's Braille literacy level and learning
styles. We suggested starting with the Animal Name Game, then Hangman, and finishing with
the Word Scramble, to have gameplay that got increasingly hard. But, ultimately, parents
played however they wanted. After participants played the games, we conducted an interview
to gather feedback on the system and learn more about families' current Braille practices
and experiences with Braille tech.
In the Findings, we look at guidance techniques that parents used with their child, how parents
adapted the games, what skills were built during gameplay, participant engagement and
feedback, and concrete improvements to the system.
So one theme that emerged from our studies is that parents often used physical guidance
with the system. Parents took an active role in helping their child find and feel the blocks.
Parents would place their hand over their child's hand and feel each hole in the block
while counting the numbers of the Braille cell. They also did this to orient their child
with the distance between blocks as well as the cardboard frame. The image on screen shows
one of our participant families using the hand-over-hand technique to feel the holes
in the blocks.
While we provided general instructions on how each game could be played, parents took
control and adapted the games for their child's grade level, Braille knowledge, and skills
that they wanted to work on. Particularly, parents used the system to build learning
skills beyond recognizing and making Braille letters. For example, parents used the Animal
Name Game to practice spelling. Two of the families used Hangman to practice name spelling,
which they identified as a key skill for Kindergarten and First Grade. Parents use the blocks to
help their child work on phonetics and sounding out full words.
Parents often used the games as guessing activities and provided hints to help their child deduce
answers. So during Hangman, parents would provide their child with hints like "This
is the name of a dog we know" or "This is your favorite food" to encourage their child
to guess through different words, rather than guess letters like the traditional Hangman
game. In Word Scramble, parents similarly provided hints for the words their child could
not unscramble on their own. The hints here took on a fill-in-the-blank form, with parents
asking "where are you blank"for the word "at" and "you drink hot blank" for the word "tea".
Many parents and children reported Hangman as their favorite game.
Parents and children were entertained by the audio feedback and sound effects provided
by the games. For example, they would laugh every time they heard the incorrect buzzer
in Hangman and the children kept screaming "it sounds like a fart!" Children would guess
silly things like "3!" so that their parents would play the sound repeatedly. Parents would
also add in their own sound effects, especially in the Animal Name Game.
So beyond learning skills, we found that children just liked playing with the blocks. While
their parents filled out consent forms or during the interview, children would build
fun structures using the tangibles. For example, the participant in the image on screen built
a mansion using the blocks, with the pegs as people living in the mansion. The people
took the train to work everyday, where the train was the cardboard frame and work was
the overhead webcam. It was really encouraging that children were motivated to use the system
to supplement their learning with creative play and storytelling.
So we asked both parents and children about their engagement while using the system, and
the engagement among parents tended to be pretty high while engagement among children
seemed to vary depending on how comfortable the child was with Braille. There are emojis
on screen representing parent and child engagement on a scale of one to five, with five being
the most engaged. Four parents reported the highest engagement of five, and one parent
reported a 3. Three children reported a five, one child reported a four, and two children
reported a two.
So parents told us their thoughts on what they appreciated about the system, and one
of them is that they felt like the system supported dual readers. One parent noted,
"having things like this where you can see the Braille and feel it is really helpful.
I love the idea for kids who still have enough vision to look at the Braille and feel it
to build their visual memory." Parents also liked the adaptability of the system to where
their child was at and even thought it could be used again for Grade 2 Braille. Parents
thought that BrailleBlocks was a fun toy option for visually impaired children. One parent
was saying, "we are always talking with other parents about what to get our kids for Christmas.
Like other kids, you get the Uno cards. But this would be great."
So we received concrete feedback on the prototypes themselves, to improve learning. Parents suggested
adding a tactile marker on the block to orient the top and bottom of the block. They also
suggested more audio feedback for children so that they could play independently. For
example, the check word feature, rather than just showing the dialog box to parents, parents
suggested including an audio description of what the child built. Alternatively, they
suggested that the system could provide minimal feedback through sound effects indicating
when a child had a letter or word incorrectly or correctly, kind of like Hangman. Finally,
parents suggested generating student progress reports to help track learning with the system.
From the Findings, our main takeaways were that participants used the games to build
Braille skills like letter recognition and identifying which dot numbers make a letter.
Participants used the system to build learning skills beyond just Braille, like spelling
and reasoning. And children used the tangibles for creative play and storytelling. Finally,
participants were entertained by the sound effects and audio feedback in the system,
and expanded on the audio by creating their own sound effects.
In our future work, we will look at how we can adapt BrailleBlocks for classroom learning.
We will work with teachers of the visually impaired to identify literacy learning outcomes
and we will also look at how the system can be played in groups of peers and monitored
by educators. Since learning has started to move online, we will also explore how to build
tools that support remote classrooms for visually impaired children as well as enable virtual
collaboration and play the system, like with sighted grandparents over Zoom.
Thank you for taking the time to listen to this work. I would once again like to thank
my co-authors and advisor Shaun Kane, my labmates in the superhuman computing lab for their
support, and our participants for collaborating with us on this project. I have linked all
of our twitter handles in this slide so if you have any questions or comments, feel free
to tweet at us or direct message us. Thank you!