[MUSIC PLAYING]
NIMER JABER: Hello.
My name is Nimer, and I work on TalkBack, which is
the screen reader for Android.
I'm here to tell you about our new braille display
support in TalkBack, debuting in Android 13.
A refreshable braille display is an electromechanical
peripheral, which is used to display
braille patterns by raising rounded pins through holes
in a flat surface.
Braille-literate computer users use the braille display
to touch-read braille dots representing text.
Users can connect their braille display to a mobile device
so that they can read and control it.
Prior to Android 13, users connected their Android device
to braille displays using the BrailleBack
app, which requires a separate download from the Play Store.
After hearing from users that they want TalkBack to support
braille displays directly, we are happy to announce that
Android 13 comes with out-of-the-box braille display
support in TalkBack.
This is the redesigned follow-up to BrailleBack,
with a new connectivity module, improved navigation,
additional commands, and various other improvements.
And we have incorporated editing via the braille display
into the existing braille keyboard input method
so that you can now use the braille display
and your phone's touch screen to put in braille text.
Braille display allows you to access
the same features of your phone that you can already
access via TalkBack.
Now I'm going to demonstrate this
by sending an email using the braille display.
First thing I'm going to do is use the panning buttons
to find the Gmail app.
And I'm going to use the Cursor Routing key to activate it.
I'm going to hit the command to go
to the top of the window, space with dots 1, 2, 3.
And I'm going to use panning buttons
to get to the Compose button and use the Routing key to activate
that.
And I'm in the To box, so I'm going
to type in the recipient's email address.
And then I'm going to use the Next Item
command, which is dots 4, 5, 8, to get to the subject.
And once I'm in the Subject field,
I'm going to hit the Routing key.
And that's going to allow me to begin typing a subject.
So here we go.
Hello from Google.
Oh, I need to fix that-- from Google I/O.
And then I'm going to hit the Next Item command again
to get to the Compose and use the Routing key.
At that point, I can begin typing the email.
Sending an email from Google I/O.
And I'm going to hit the command to go
to the top of the screen, which is, again, 1, 2, 3 with Space.
And then I'm going to use the panning key
to get to the Send button and activate it
with the Cursor Routing key.
OK.
Thank you for your time today.
Please look for braille display support in TalkBack
coming in Android 13, coming soon.
SCOTT ADAMS: Hi, I'm Scott, Product Manager on Lookout,
an Android app that describes the physical world to people
who are visually impaired.
I'm here to discuss image understanding.
Images are rich sources of information.
Photography, graphs, illustrations, cartoons, memes,
visual art--
they all help us communicate.
For people with little or no functional vision,
that content can be impossible to access if there's
no alt-text or caption.
Imagine reading a news story with a photo
or receiving an image on social media
that people are reacting to, but not
knowing what's in that image.
That's where the potential of ML comes in.
What if a model can describe that image for someone
who's blind or low vision?
It's an exciting area, but there's some complex questions
to consider.
Here's how we set out to design and build an ML-based solution.
We began with our users.
We sought to understand what people
who are blind or low vision wanted to learn about images.
The guidance we received was, give us all the information
you possibly can.
There was no one subset of information
that was universally helpful.
What's interesting to one person in one context
may not be interesting to another person
or in another context.
As a result, people who are blind or low vision
told us it was best to describe everything in the image.
But accuracy was also very important.
Mistakes can be confusing.
That led to complex technical questions.
First, we realized we couldn't do everything
that users asked for.
There was no universal model that provided complete detail
across all image categories.
So we had to limit the range of images we could interpret.
Our goal was to pick an image type that
was broad enough to be useful but narrow enough
to be tractable.
We also had to consider what information to present.
What was the best way to describe an image
without being overwhelming?
Which elements could we reliably detect with high accuracy?
Finally, we had to think about the model outputs, especially
if an image contained people.
We wanted to ensure the results were respectful, safe,
and fair.
So what did we decide and how?
To begin, we engaged with the AI Principles, Product Fairness,
and Ethical AI teams at Google.
We performed extensive user research, not just in the US,
but around the world, as well as a fairness analysis
and adversarial evaluation.
Moving on to the details.
Starting with the type of images,
we decided on photographs-- in particular,
photos you might find on news sites or social media sites.
This covered some common use cases.
And we also had access to large, diverse training sets.
Plus, in some cases, a user could verify the model's
result. What does that mean?
So imagine that a user has an image with a caption,
and the caption is generated by a human.
They could then use the machine to generate a caption that's
been automated and compare the two to see
how the human caption and the machine-generated caption
compare.
Moving on to technical constraints.
While people wanted both high detail and high accuracy,
accuracy was more important.
So we favored precision, speaking results only
when the model was quite confident.
In terms of the visual elements to report,
we learned it was best to provide a summary for context
and then extra detail where we can.
So for example, if the image contains people or text,
those can be especially important to describe.
Finally, we also had to consider how to present
the results to the user.
For some types of descriptors, we
had to determine how, or if, we could provide
that information respectfully.
Here's what I mean.
Consider hair length.
Imagine a model can detect long hair or short hair.
That's subjective.
Different people have different ideas of what each means.
So a better response could be shoulder-length hair.
We also wanted to ensure that the results weren't hurtful.
For example, the term "bald" could
be hurtful to some people with no hair.
Since no model is perfect, we also had to think about errors.
This area was challenging, but we
aimed for results that were both objective and observable.
It also meant that for some descriptors like hair length,
we decided we have more work to do.
That's how we approached the problem.
So what do we build?
Lookout's new Images Mode can do the following.
It can offer a caption for an image.
For example, "Dogs playing in the grass" in the first image.
If people are present, it can describe each person's
upper body clothing and whether they're an adult or youth.
For example, for one of the people in the second image,
"Youth, black shirt."
It can label details in the image.
For example, "Hills, clouds."
And finally, it can read any text that's present.
So, "Coffee, croissants" in the fourth image.
All this works on-device with no network connection needed.
We're releasing Images Mode as a beta on June 2,
with some features available in English only.
We hope to fully support more languages soon.
We know we're just scratching the surface.
We want to provide still-richer detail with greater accuracy.
We hope to improve our models over time
and explore more ways of understanding images.
We'll continue to work with our user community and the AI
Principles teams to ensure what we provide is responsible,
useful, and accessible.
If you're pursuing image understanding technology,
here are a few things that we've learned.
As with anything accessibility, the foremost guidance
is to engage with the community early and often.
This helps to ensure that what you're building
is something that people will find truly useful.
Getting into the details now, the range of image types
is vast.
So constrain your solution to something
that feels approachable.
When training models, use the largest,
most-diverse data sets that you can.
As for the elements in an image, consider
offering both summarization for context and then detail
for richness.
We also found that accuracy is especially important.
So consider a high bar for precision,
and then explore ways to improve recall.
As for the results themselves, focus
on what's externally-observable and objective.
Also imagine cases where the model is right or wrong.
How might someone experience both?
Invest the time to understand the risks of harm
and opportunities to favor respectfulness.
Finally, this area is rapidly developing,
and there are still many, many limitations.
So set expectations with your users.
Thank you again for your time.
And if you or someone you know has
a substantial visual impairment, we
hope you try Lookout from Google,
available in the Play Store.
Thank you.
VINCE WU: Hey, everyone.
I'm Vince, Product Manager working on Android
accessibility.
You heard earlier from Nimer and Scott
about TalkBack and Lookout, two services that help people
with vision disabilities.
First, I'll be talking about a new, additional service we're
designing for to help people with vision disabilities
read long-form content.
Let's go ahead and get started.
On the user problems, I'm reading,
and I really want to improve my ability
to see and to comprehend.
So I might want to increase the text size,
increase the line, the word, the sentence spacing,
maybe the color contrast.
What I really want to do is to eliminate the distractions.
And in some cases, it would be really great
if I could hear it read aloud.
Accessibility Reader is a new experience
that helps people with low-vision disabilities
read longer articles.
It's universal.
It works across screens, both on the web and native apps.
It's on-device, meaning that it's low latency.
And it's privacy-sensitive.
It transforms reading into accessible, simple,
and focused experiences for easy reading or audio playback.
So I can increase the text size.
I can change the line, the word, the sentence spacing,
and the color contrast.
I can read the content myself or hear it read aloud.
Next, I'd like to talk about services that help people
with hearing disabilities.
First is Live Transcribe and Sound Notification.
This is a service that provides real-time speech-to-text
transcription as well as notifications
about surrounding sounds.
With real-time transcription, Live Transcribe
does speech-to-text in real time.
So the text appears on your phone as words are spoken.
The service already supports more than 80 languages
and dialects.
And you can quickly switch between two languages.
What's new is the offline mode, meaning
real-time text-to-speech transcriptions that
work even when you're not connected
to Wi-Fi or a mobile network.
This is now available in English,
Japanese, German, French, Italian, and Spanish.
Sound Notifications uses on-device machine learning
to notify you through haptics, flashing lights,
and phone notifications when there are important sounds
around you--
for example, smoke alarms, sirens, baby sounds.
You can then be notified with a flashing light
or vibration on your mobile device or a wearable.
The timeline view lets you go back in history
and see what has been happening around you.
What's new is personalized sound detection.
You can customize and train to your specific household sounds.
So, for example, I know my food is
done when I get my microwave beeping notification
versus the dishwasher beeping.
The final service I'd like to talk about is Sound Amplifier.
It's a service that helps enhance audio
from the world around you or the media playing on your device.
We continue to iterate and learn improvements that
make the experience better.
We're coming up with a brand new UI.
And we're also working on better noise cancellation as well.
Now with that, that's the end of the session.
Thank you for tuning in.
My name is Vince.
See you.
[MUSIC PLAYING]