Hi everyone. Today I'm going to give the
presentation for the paper Assistive-
Technology Aided Manual Accessibility
Testing in Mobile Apps, Powered by Record-
and-Replay.To get started with, we know
that nowadays, software and digital
technology are part of humans' daily
lives, especially mobile devices.
Billions of people use them on a daily
basis. With a few taps,
we can order food from Doordash, we can
connect friends from Facebook, we can
book flights from Expedia, and we can
look up the word definition inside the
Dictionary app. However, among all
the mobile device users, we often
overlook people with disability.It's
estimated that around 15% of the world
population have some form of disability.
Recent studies they just found many
popular apps are shipped
with accessibility issues.Uh, no
exception for those four.So why mobile
developers will create inaccessible
apps? The following slides better
illustrate the challenges mobile
developers are facing.So first
of all, previous study found mobile
developers they still prefer manual
testing during the development, so they
will use touch gestures to verify the
correctness of their functionalities.
For example, we're inside this budget
tracker app and we want to test whether
this add button next to salary or wages
will work or not. So as
a developer we're going to use our
finger to single-tap on this add button.
And after tapping,there's a
pop-up window appears to inform us to
type some numbers for salary and wages.
After seeing the result, developers
might think, OK, this meets my
expectation. There's nothing wrong with
this add button, so I'm going to test
the rest of the apps.The thing
is, people with disability, they usually
can't use touch gestures. Instead, they
rely on assistive technology or AT
to help them interact with the apps.
And there are various assistive
technology available in the market,
TalkBack, Switch Access and Voice Access.
And each assistive technology has
different interaction modes. Take TalkBack
for an example. So blind
people will use TalkBack, the Android
screen reader, to help them navigate
through mobile applications.
So let's see how blind people use
TalkBack uh to interact with the same
app. The difference is that blind
people, they cannot see anything on the
screen, so they rely on TalkBack
to go through elements one by one,
and they rely on TalkBack to
announce the textual description of the
element so that they can determine
whether this is the element they want or
not. So we're gonna start
exploring the screen by using TalkBack.
So we firstly we do a swipe right
and we focus on the element
and TalkBack announced Select
Categories, so we know it's not the one
we want and we do another swipe
right and this time TalkBack, focused on
Disposable Incomes and announced it.
Still not the one we want, so we might
need to do another swipe right
and TalkBack focsued on a different
element announces as Add New Category Button.
Sadly, this is still not the one
we want, so we might need to do
an additional swipe right.
And this time, luckily, TalkBack focused
on the element and announced Add Button
to us, so we know, OK, this is the one
we want. Once located the element,
so we're going to double-tap on the
focused element. That's the equivalent
action for the single-tap when we use
the touch gesture. So let's see what
happenes.
Oh, nothing happens here. So after we
use the double-tap for the focused
element, nothing happens here. So
what we expect here is we want to
see a pop-up window to appear,
but there's, that's not the case for
using TalkBack, so that means there's
an accessibility issue associated with this
add button.So why mobile
developers will ignore those kind of
issues? It's because they do not
have enough knowledge of assistive
technology. At this point, some people
might argue, OK, I know this time manual
testing does not work. how about, you
know, existing automatic accessibility
testing tools? For example, Google
Accessibility Scanner. OK, let's see
what kind of results Google scanner can
give to us. So we scan this
add button and scanner provides
three suggestion to improve
accessibility, for example, to make the
add button much bigger,
to make the contrast much better
and we can see that the Google scanner
does not consider the use of
AT. So that's why Google scanner cannot
report the
issue associated with add
button to the developers.
Considering the current situation, so we
would like to develop some tools,
to still allow manual tester or
developer to assess their apps using
touch gestures, while we can detect
accessibility issues inside their apps.
The detection of accessibility issues
should not require prior knowledge
of AT from developers.
Based on those insights, we
proposed A11yPuppetry,
to aid developers in gaining insights
about accessibility issue inside their
apps. So our tool employs
record-and-replay techniques and there
are four phases inside our tool. So
firstly, we're going to record a touch
gesture from developers and translate
those gestures to the TalkBack actions
and replay the TalkBack action in a
different device. Finally, we're gonna
analyze the execution result and
generates visualized report to the
developer.And move on,
this slide better illustrates how our
tool will work.
So, again, firstly,
developer can assess their apps using
touch gesture manually while
our recorder will record those
touch gestures. And
after that, we will send the result
from the recorder to the server
and at the same time, we will translate
the touch gesture to the equivalent
TalkBack actions.
After receiving the TalkBack actions,
TalkBack replayers inside our tools
will replay them in a different device
and send the result to the server.
Finally, our tool will analyze the
execution result and generate visualized
report to the user, to the developer.
So to evaluate the effectiveness of our
tool, we conduct user studies. So we
selected 5 apps with confirmed issues.
And for each app we designed
task, consisting of 20 to 40 actions
and then we used Fable Platform
to conduct compatibility test
and user interview. So during the test
and interview,TalkBack user
will perform our designed task and
report any of the accessibility issue
inside those tasks. So the reported
issue will be regarded as a benchmark
for our tool. So then we will
use A11yPuppetry to record-and-
replay the task and to see what kind
of issues our tool can automatically
detect. And finally, we compare
our tool with existing tool
Google Scanner and Latte.
So through the user study, we found that
our tool was able to detect uh many
accessibility issues automatically. For
example, unlocatable element.
So this speaker button, annotated by the
pink box cannot be focused on by
TalkBack. TalkBack will just ignore this
speaker button. So our tool
was able to detect those issue
automatically. In addition,
the ineffective action issue, that's the
one we saw in our motivation example.
So after we double-tap on this add
button, nothing happened here, so our
tool again, can detect those issues
automatically.And finally,
we compare our tool, the
performance of our tool with existing
tool-Google Scanner and Latte
and the comparison is based on the
detection rate of the user confirmed
issue. So overall, by looking at this
table, our tool is able to
detect over 60% of the user confirmed
issues and outperforms
Scanner and Latte.
In conclusion, this presentation
introduced a novel tool
called A11yPuppetry to aid developers
in gaining insights about accessibility
issue inside their apps. Our tool will
record touch gestures from
developers, replay the
equivalent TalkBack actions, and
generate visualized report to the
developer.Through the user study, we
found our tool was able to
detect ineffective action and unlocatable
element issues automatically.
So, that's the end of the presentation.
Thanks for your listening.