Hi, everyone, my name’s Tom Hume. I’m a Product Manager in Google Research.
I’m here today with my colleague Scott and Sagar,
to talk about the intersection of machine learning and accessibility.
Around the world, 1 billion people currently experience some form of disability,
which means people with disabilities are using your app right now,
and in larger numbers than you might expect.
Thanks to advances in machine learning, today we can help people in ways
that just weren't possible a few years ago.
So, today we'll show you a few examples of this help,
tell you how we’re thinking about accessibility and machine learning
and give you a few hints for how you might apply it in your own apps.
As it happens, I’m the product manager for Voice Access.
So let's start there.
What's Voice Access?
It’s an Android app that lets you control your phone using your voice.
We designed it for and with people with manual dexterity challenges,
but we think it’ll benefit all who wants a hands-free experience.
Once you’ve downloaded Voice Access from the Play Store and set it up,
you can give your phone simple commands.
Commands like open Maps, search for Stinson Beach,
tap directions.
We launched a big redesign of Voice Access last year with a huge set of improvements.
We rethought the user interface. We added some new capabilities.
Today, I’d like to talk about how we use machine learning
to make Voice Access easier and faster.
So, let’s step back for a second and remind ourselves how accessibility works on Android.
When you’re writing an app, all the different components of your UI
have some text associated with them.
This text is used in different ways.
A screen reader for the blind might read it out.
Voice Access uses it to provide labels on screen and to understand commands.
If you're using a good user interface toolkit,
this text on some other important information is automatically set up.
But for some elements like photos and icons, developers need to add the labels themselves.
I’m sad to say that many developers don't have these labels
or use toolkits that don't support accessibility well,
which means important parts of their apps are just invisible to their users.
There’ve been studies into this.
If you Google inaccessible button disease, you’ll find a good one.
And sometimes, even when developers put in the work, they focus on screen readers
and text that works there isn’t so good for Voice Access.
But, what if Voice Access could look at images on screen
in the same way a sighted person would recognize icons and give them labels?
Well, this would have two benefits.
Firstly, where an app developer
hasn't given an icon a label, Voice Access can add one,
but also it would mean that users could refer to the same icon
with the same name consistently across apps.
Our classic three dot overflow icon, for instance,
might be labeled Menu by some apps, Overflow or Options by others.
Now, all these apps are trying to do the right thing by their users,
but we shouldn’t expect users
to learn different names for the same icon in different apps.
So that's exactly what we did.
Android R added a new screenshot API for accessibility services to use,
Voice Access users this new API to take a screenshot
and passes that screenshot into a machine learning model we call IconNet.
IconNet gives precise information about which icons are on screen and where,
giving them labels and then Voice Access takes those labels,
plus the ones an app has provided and uses them.
And here's my favorite part.
It does all of this locally without your screen ever leaving your Android device.
We think this is a great use for machine learning to fill in some of the gaps
in Android apps for users with accessibility needs, and to do so quickly and privately.
Voice Access detected about 30 icons when we launched.
We added another 40 in February, and there’s more intelligence to come.
I’ll end with a plea to developers,
please download Voice Access from Play Store
and use it to test your applications.
There are literally millions of people worldwide who have manual dexterity issues.
Voice Access gives them full use of their Android device and your apps.
Testing your app with Voice Access is good for them, and it’s good for you.
You can download it at g.co/voiceaccess.
And now my colleague Scott is going to talk to you about Lookout.
-Scott. -Thanks, Tom.
I’m Scott Adams and I work in research as the product manager on Lookout.
And I'm here to talk to you about a case study
in ML applications for people with visual impairments.
Lookout is an app that uses the smartphone's camera
to recognize objects and text for users who are blind or low vision.
We use cool technology that’s available to you through ML KPIs.
But there’s more to it than connecting a camera to a classifier.
Especially in accessibility it's critical to design
with your users and not just for them.
There are two questions to ask.
One, how well do I understand what the user needs?
And two, how can I fit the technology to those needs?
At Google, this is codified in our AI principles such as:
be socially responsible, and be built and tested for safety.
For example, Lookout went through proactive adversarial testing
to guard against unfair bias.
This kind of sophisticated evaluation is critical,
but the path is much easier if you build it with users from the very beginning.
For instance, Lookout uses several different image classifiers.
Here’s how I used user feedback to tune for precision and recall.
As a refresher, imagine that we have a mix of apples and oranges
and an apple detector.
If we have a high-precision apple detector, then the detector is taking no chances.
It’s only going to say apple if it’s certain I have an apple in my hand.
On the negative side, if I have an apple that’s, let’s say, oddly shaped
or a different color than usual,
it’s going to say nothing, so we’ll have false negatives
where I do have an apple in my hand but the detector is silent.
Contrast that with high recall.
In this case, the detector may be so sensitive
that anything that is round and about the size of my hand is an apple.
So, every time I show it an apple, it’s saying apple every time.
That's terrific.
But the downside is, I might show it an orange,
and it’s going to say apple, and that’s a false positive,
where it’s saying apple but there is no apple in my hand.
And ideally, we have both high precision and high recall,
but that’s often not possible, so how do we tune that?
Well, it may come down to the user.
If I really prefer an apple to an orange, I may prefer a high precision detector.
With that in mind, here’s what we learned from our users on Lookout.
I’ll talk about two cases: one on currency, and one on objects.
So for currency, some types, like US dollars,
are the same size, color and texture regardless of their value,
which makes it impossible to distinguish
a $1 bill from $100 bill if I can’t see the bill.
So our goal is to identify the value of the bill.
Now for objects, imagine that I can't see
and I'm going into a room that I'm unfamiliar with.
I might want to know what’s in that room, what kind of furniture,
so I can understand what kind of room it is, like a living room versus an office.
So with that in mind, what did users tell us?
For currency, it was fairly intuitive.
They said, "Listen, never confuse a $1 bill for a $100 bill.
Don't guess. If you're not certain say nothing."
So in this case, this is a pretty clear signal for a precision.
Now, the easy thing to do is at first glance take that lesson and carry it over to objects
but there's some nuance here.
Imagine we have an object, and it could be an armchair
or it could be a couch.
If we're very high precision, we may say neither because we’re not sure.
And from users, we learned that actually either answer has some value,
even if it’s imprecise.
So whether it's an armchair or it's a couch,
I know that this is a pretty big object and this room might be a living room.
So we actually wanna balance between the two here.
So taking a big step back,
getting this user feedback about development is critical.
Do it during design, during development, during testing, not after release.
It takes longer but you make a better product.
Okay, if you’re excited to start writing
your own accessibility apps with computer vision, then dive into ML Kit.
We have barcode scanning, OCR and object detection.
And if you or someone you know is blind or low vision,
please consider Lookout.
Next, I’m pleased to introduced my colleague Sagar.
Hi, I’m Sagar.
I work in Machine Perception team in Google Research.
My team's mission is to help machines understand the world like humans do.
We work on Live Transcribe and Sound Notifications.
Live Transcribe is an app for deaf and hard of hearing people
to get captions for real-time conversations in over 80 different languages.
Live Transcribe is a great example of using ML technologies
together with good UX research to provide meaningful experiences.
It uses automatic speech recognition, of course,
as well as models to detect speech versus other audio
and also a sound event detection model to provide sound chips to the user.
I want to share this story of how Live Transcribe came about.
Meet Dmitri, a computer science researcher
who has been working on speech recognition for over three decades.
Dmitri typically relied on lip reading and professional captioning for communications.
As good as he is with lip reading,
we would often struggle to have impromptu conversations
around the water cooler.
Lip reading alone is only about 50% accurate
and professional captioners are not always present.
As ASR technology improved and became more pervasive,
Dmitri launched an experiment with the team to create an app for real-time captioning.
That Android app uses Google's Cloud Speech-to-Text API
to turn that water cooler chat into captions on the screen off his phone.
Together with Dmitri’s decades of experience in speech technology,
the team iterated on the app and improved it to a point
where he started using it daily for personal and professional conversations.
Dmitri was excited to try more things in the app.
Motivated by his insights as well as enthusiasm,
we started looking at what we could do to make the app better.
What if the app could interpret more than just spoken conversation?
A few years ago, we released a data set called AudioSet,
which allows developers to recognize over 600 different sound event classes
in their applications.
The easiest way to get started with AudioSet
is to start with the pre-trained model from that dataset.
For that, you can directly integrate
our open-source pre-trained model called YAMNet
to understand different sound events.
Here is a use case demonstrating how critical this non-speech information can be.
Dmitri recalls a story of how one day he was asleep
and the smoke alarm was active.
He could not hear it,
but luckily his neighbor came in and woke him up.
As our researchers talked to more potential users,
we heard countless stories like this
and this led us to use Haptic Alerts, vibrations delivered to a smartwatch
to notify users about important sounds in their home.
So a few months ago, we launched our Sound Notifications.
It tells you when it hears sounds around your home,
like sirens, dogs barking, babies crying, water running
or if somebody knocks on your door.
We also open-sourced Live Transcribe Android app engine,
so developers can customize it for their own use cases
by integrating it within a bigger existing app
or porting it to other platforms.
Another search experiment of using ML for accessibility is Project Shuwa.
Shuwa, which means sign language in Japanese,
is a project centered around sign language detection and understanding.
Together with the Nippon Foundation and the Chinese University of Hong Kong,
we created a web game for people to learn a bit of Japanese
and Hong Kong sign languages.
Since there are over 150 different sign languages in the world,
including American sign language, Indian sign language, and many more,
to better help developers create their own gesture
and sign language understanding systems,
we have open-sourced our gesture detection toolkit for developers.
You can check out demos of Project Shuwa at the I/O sandbox.
Thank you for watching.
Just like Dmitri and our team’s prototype Live Transcribe and its extensions,
with many existing bits and pieces,
we welcome developers to experiment with such different machine learning techs
to help the world become more accessible.
Please reach out if you’d like to learn more.