Hello, my name is Tim.
This video is a replacement for me and my colleagues' CHI 2020 presentation, which was
supposed to be given in person in Hawaii. As I'm sure you know, CHI 2020 was cancelled
due to the COVID-19 crisis. This talk is about our paper CreaTable Content and Tangible Interaction
in Aphasia, and reports research undertaken by HCI researchers and speech and language
researchers at City, University of London. In this work, we have co designed a tangible
platform and media sequencing app called CreaTable with people with aphasia. Aphasia is an acquired
language impairment which sometimes follows a stroke.
Creative acts are often evocative,
they might allow us to convey thoughts, emotions, or rich imagery. They enable us to express
ideas we might not otherwise be able to express through traditional communication routes,
such as speaking and writing. Further to the expressive benefits of creativity, being creative
has also been shown to afford improved mental well being and self esteem. Sharing digital
media online is now commonplace. Much of this content is multimedia, allowing for multiple
modalities including non language modalities. And the creation of multiple types of media
offers an opportunity for people with language impairments to be expressive in new ways.
However, effectively engaging with digital technologies to create such content can be
challenging.
The tools we use to create such content
generally require complex, language laden interactions. This poses a significant barrier
to access for users with aphasia.
Tangible interactions,
ie physically interacting with digital data are a potential means to address this challenge.
In this presentation, I will describe CreaTable,
a tangible platform and media sequencing app designed with people with aphasia. I will
provide some background, describe the tool itself, the co design process and the use
of CreaTable in a digital content creation workshop.
First, some background.
This work sits at the intersection of tangibles for content creation, and content creation
for people with aphasia. Previous work has investigated how tangibles might be used to
support creative endeavours. A common example of a tangible input device for drawing is
the stylus. However, a number of prototypes have been built which offer a range of controls,
which offer more diverse manipulation. Reactable is probably one of the more notable examples.
Reactable will afford collaborative composition of music on the table using puck-like tangible
objects. Reactable is just one example of a tangible system. See the paper for more
details and a comparison to our work. Aphasia here is an acquired language impairment. It
is most commonly caused by a stroke. It can be caused by other forms of damage to language
regions of the brain. It can affect reading, writing, speech and comprehension. The aspects
of language affected can vary significantly between people. Some people might be only
able to produce a few words, but retain good levels of auditory comprehension. Against
this context, we present a tangible platform - CreaTable, and a media sequencer app that
we've developed to support people with severe aphasia in creating expressive multimedia
digital content.
CreaTable is a tangible platform for
creating and curating multimedia content. It consists of a table - where tangible objects
are arranged by users. A webcam and fiducial based recognition system detect the position
of objects to create an equivalent digital representation. CreaTable can be used in a
number of ways. In this paper, we describe the usage of CreaTable as a media sequencer
CreaTable is 780 millimetres in width
490 millimetres in depth
715 in height. Top surface is 10 millimetre perspex mounted within an aluminium extrusion
frame. a webcam below, tracks fiducial markers on the top surface using the Reactivision
framework. Tracing paper is used so that users cannot see down, but the camera can see the
markers on top. Users place content represented by tangible tokens on the CreaTable. These
can be pictures, words, recorded music, or even individual notes.
Content is represented by an "Arrange" display.
The Arrange display presents tangible tokens on the screen in digital form. This display
updates as users move content about the surface of CreaTable. The content is then played with
the knob on the front of the device. The user controls the playhead speed by twisting it
from left to right. The playhead speed can be positive that is going forwards or negative
going backwards. Its speed can also be reduced to zero -that is stopped. The playhead is
represented on a range display, and also via a programmable LED strip embedded into the
frame of the device. One of the core design aspects of CreaTable is this looping feature.
When the playhead reaches either end of the CreaTable it loops back to the other side,
allowing repeated playing. The visual content is then displayed on the play screen, audio
through a speaker. CreaTable was co designed with four people with severe to moderate aphasia
over the course of three, two and a half hour sessions. In workshop one we explored the
notion of tangibility. People with aphasia often have difficulties using their right
hand or arm. We wanted to understand what opportunities and challenges different sizes
and shapes of tangible controls best supported people with aphasia. We explored tangibility,
through hands on exploration with early prototypes of music manipulation and image manipulation
via blob detection using an overhead webcam. Tangibles were received positively and looping
was found to be a useful approach as it allowed people to iteratively refine their idea. In
workshop two, we focused on a range of different media types. Specifically, we focused on the
types of content that people would like to create. From this, a mixture of complimentary
media emerged as an interesting theme. We also discussed the potential for cutting down
the amount of content. The one to one mapping between digital and physical content meant
that co designers are often overwhelmed by the amount of content on display. In workshop
three, the final workshop, we built upon these ideas, and thought about how we might convey
different sentiments and moods. We refined the prototype and protocol again. For example,
we now have the initial version of the prototype within an aluminium frame with a camera looking
up from below to the surface. We explored the idea of rotatable pentagonal tokens which
allow the user to access a group of content as they are rotated. We refined the design
based on feedback from a final co design workshop. We then ran a digital content creation workshop.
We wanted to understand the usability and the possibilities of CreaTable for enabling
people with aphasia to create digital content. 11 people with aphasia participated in the
workshop using CreaTable to convey specific ideas and sentiments. Participants were split
into three groups.
Each chose a
random theme by choosing pairs of words on a piece of paper, for example, "Happy" and
"Earth". Content came from a range of sources, some of which were created by the co designers
themselves. Three groups made a range of different types of content. Group three, who I'll focus
on for brevity, chose the words "Calm" and "Wind". They chose one of the co designer's
paintings, choosing the word "wild" and "angry" to match it. They then selected a picture
of a solitary fisher, adding the word "quiet". Then match the word "strong" to a pair of
skiers climbing a mountain. Then finally used a picture of the sea to match the word "love".
They explored a range of music by playing different tokens, finally deciding on a mellow,
soft piece of acoustic guitar music.
Here's the final composition.
We took a range of subjective (such as likert data and spoken feedback), and objective data
(such as analysis of interactions and the content created). I'll focus more on the subjective
data for the purposes of this presentation. The post study likert results generally indicated
a positive response towards CreaTable. Most participants felt they made something new,
that they couldn't have made without CreaTable. Although many participants expressed that
they enjoyed using CreaTable in a group, and that they'd like to use it again, many also
wanted to try using the tool alone. Some participants noted that this was due to tensions in the
group nature of the creative process. Participant 11 said, "Challenging to create something
with others when their choices don't reflect or gel with mine." On the other hand, we've
received some positive responses from people who'd made content in a group. Participant
9 said "It's nice to have people around you have different opinions." Participant 10 said,
"Yeah, didn't mind. You choose yours, I choose mine. Okay." Some participants noted positive
sentiment towards sequencing the content. Participant 10 expressed "Strong. Determination.
Keep going." In general, the participants liked the one to one mapping of the prototype.
Participant 9 noted, "You just choose what you want
and put it on the table. Simple."
CreaTable sparked discussion in all groups. Notably, people used it to tell stories about
themselves, or match the content they were creating to other media or other world events.
One example of this was participant 11 she noticed the word "shock". In dark humour.
She responded "Shock? Having strokes is shocking, isn't it?" She then rotated through the different
words, "Anger, having a stroke made me angry as well". Rotating to "cold" she then went
on to describe an experience of a movie she saw set in Russia, which made her feel cold.
The data from the creative workshop suggested that participants with a range of aphasic
difficulties were able to engage with CreaTable, that the tangibility of the prototype afforded
them the ability to create expressive digital content. While we saw some tensions in group
use, the participants felt that the content that they made was theirs. This suggests that
the group dynamic did not affect ownership, and that users did not feel that the tool
was automating some of the creative process, a common issue in creativity support tools,
we saw CreaTable being used for far more than creating expressive content. We saw it being
used to support challenging and humorous discussions about lived experience. This indicates some
potential in using tangible content creation activities in stroke groups.
To conclude
with increasing numbers of people acquiring stroke related language impairments, and the
ever growing power of digital content. It is essential that content creation tools are
made more accessible. Here we have presented a tangible content creation prototype called
CreaTable. CreaTable has allowed us to think about how content creation processes and tangible
interactions might be better adapted to support accessible creation of digital content. Our
work shows the benefit of such approaches in supporting people with aphasia in creating
digital content and the merits of engaging with people with aphasia in the design and
testing
of content creation tools.
Transcribed by https://otter.ai