REBECCA NATHENSON: Hi, everyone.
I hope you're enjoying your time with us
today at Google I/O. Welcome to this technical session, where
you'll learn how Google Assistant's architecture powers
voice features in your apps.
I'm Rebecca Nathenson, and I head up the Product Management
Team for the Google Assistant Integrations and Developer
platform.
It always amazes me how deep the technology is behind
making Assistant work.
I think for a lot of us who grew up
with some kind of sci-fi, voice-powered assistant
on TV, whether that's Jarvis, or the Star Trek computer,
or any other type of magical voice interface,
it's easy for us to gloss over just how hard
getting this right really is.
I know when I joined Assistant in 2020,
I was amazed at the depth of thought, design, and tech that
went into a simple utterance like "Book a table for four
and 8:00."
And when we add in the ability to understand
what kinds of actions apps can take, and let the Assistant
connect users with those actions wherever it makes the most
sense, it feels like magic.
But it's magic that is backed by so much tech.
Today, we're going to walk through an overview
of Assistant's architecture and what
goes into enabling voice interactivity on so
many devices across the world.
After that, we'll review ways developers
can plug into Assistant to take advantage
of voice-forward APIs.
Thanks, Rebecca.
I'm Alec Truitt, and I work on Global Product Partnerships
for Google Assistant.
My team works with partners to explore new assistant platform
features and create exciting experiences for their users.
Before we dive into Assistant's architecture,
I'll share a brief background of Google Assistant and our vision
for where we're headed.
Back in 2016, we first introduced Google Assistant
because we wanted to help users get things done.
We knew that enabling users to engage
with Google Assistant for what they wanted by asking questions
as a natural conversation like, "Hey Google,
what's the weather?"
was the best way to achieve this new frictionless, helpful,
and personal experience.
Since then, Google Assistant has grown
as a conversational agent available in over 30
languages, 90 countries, and on more than 1 billion devices.
Whether the user is on their phone, in the car on the go,
or at home, we are developing a system
of experiences that seamlessly span
across the user's many devices and contacts
throughout the day.
Our goal is to enable cohesive, helpful, and inclusive user
journeys informed by feedback from developers like you
by developing healthy ecosystems to support these experiences.
Users don't just want a more natural way
to get what they need.
They want to get more of what they need in that natural way.
Google Assistant is natural in that users can just
say what they want and it's not tied
to a single device or service.
We build Assistant leveraging the latest advances in machine
learning, AI, and natural language understanding along
with Google's technology stack to create an advanced assistive
experience.
Our developers play a crucial part
in how we can deliver on those millions of interests
from users, such as asking assistant "Hey Google,
when is my Walmart order arriving?"
to check the status of an order.
Assistant is evolving to become a conversational agent that
helps users get things done with their favorite apps
and devices.
We've seen that users want a more seamless voice
experience and the opportunity to switch
back and forth between voice and other modalities,
such as touch.
We're excited to share the new scalable Assistant integration
paths to enable these voice-forward experiences
for users.
Next, let's explore the components
of Google Assistant's architecture
that bring these conversational assistive experiences to life,
and how you, as the developer, can
plug into these common development and testing tools
to build on this infrastructure.
Let's take a look at an example, my family
knows all too well-- planning our weekly grocery order.
Assistant can now string together
many segments of this journey, such as looking
at items we order most often, adding them to cart,
and placing the order.
Then I can simply ask Assistant, "Hey Google, book a time
with Walmart to schedule our grocery pick-up."
When a user makes a request to Assistant,
there are three important steps that happen in the background
to quickly fulfill it.
Assistant receives the user's request,
processes it to understand the best action to take,
and finally fulfills the user's request.
When a user makes the request such as,
"Hey Google, book a time with Walmart,"
Google breaks out the parameters of the query.
And then Google's natural language understanding,
also known as NLU, plays a key role
in matching it to the user's intent
while also factoring in the user's context and device type.
We think of Assistant as having a few superpowers that
can create value for users and developers.
Google's NLU is a great example of this.
And another very important one is called actionability.
This is the idea that Assistant can understand and operate
with a set of intents any given app supports.
The final step to fulfill this request
is to deep link the user into the Walmart app
to reserve a pick-up order time.
Assistant does this by understanding
what capabilities apps support and which apps can best
fulfill the user's request.
By integrating with Assistant, your app
can leverage Google's infrastructure
to power these voice-forward experiences for your users.
Now that we've covered the process at a high level,
I'm going to hand it over to Luyan
to review the special components of Google's architecture
like our evolving NLU capabilities in more detail.
LUYAN CHI: Thank you, Alec, for introducing the Assistant's
overall architecture I'm Luyan Chi, a software engineer
working on Google Assistant.
My team works on building the platform for developers
to integrate with Google Assistant.
Now let's dig more into Google super power.
That's the part that really makes your [INAUDIBLE]
integration even better.
A key ingredient of Assistant is the capability
to transcribe speech into text.
That's what we call speech recognition.
When a user issues a voice query to Assistant,
the targeted device's voice assistant support
will open a streaming connection with the backend server,
which receives audio data and converts speech to text.
We don't simply transcribe one word
by another in the middle of a dialogue.
Google Assistant biases its results
to listen for what is expected, such as a yes or no answer,
just as a person might.
It also uses information, such as the contact names,
to improve transcriptions.
This is called speech biasing.
With this, Assistant is able to transcribe words
into text accurately.
After the speech recognition step,
the Assistant platform will convert the use of query text
into interpretation, which is a formal description of what
the user wants, with the intent and arguments written
into a structured format.
As Alec mentioned earlier, this is called natural language
understanding, NLU.
Understanding intent is not to a simple task.
For example, if a user's response to a question
is "yes," that is pretty straightforward.
But what about all the variations of "yes,"
like 'that works," "sure," "why not," "I agree," "go ahead,"
and so much more Google Assistant's NLU
can understand different variations of a user's input.
Let's look at another example of natural language understanding.
The user might ask Assistant to "Book a table for four
at Bill's restaurant for tomorrow night at 8."
As so you can see, there are two numbers in the query, 4 and 8.
The Assistant's NLU is able to understand
that four means four people, and tomorrow night at 8
is a specific date and time.
Assistant NLU is also able to take context
into consideration.
For example, if a user asks, "How is
the weather in Mountain View?"
after getting a response, the user issues a follow-up query,
"How about New York?"
in this case, even though the second inquiry
doesn't mention weather, Assistant
is able to understand from the context
that the user is asking about weather
and will be able to respond correctly.
What's more challenging is this--
users talk to Assistant in a natural way,
which sometimes includes a multiple back and forths
between the user and the Google Assistant.
That's what we call a dialogue.
For example, if the user asks the Assistant
to "Book a table for four at Bill's restaurant,"
without saying when, the Assistant
replies with, "When do you want the table?"
or something similar.
The user can then say, "Tomorrow night at 8:00."
The dialogue continues until the Assistant
has enough information to complete the request, at which
point the Assistant summarizes the request,
asks for confirmation, takes action, and confirms
completion.
As you can see from these examples,
natural language understanding is a complex task.
There are many different variations
to express one's intent.
Users prefer to speak with Assistant
in a natural and conversational way.
When understanding the user intent,
we can't simply look into a single user query,
but need to take context into consideration
and be able to support multi-turn dialogue,
not to mention the complexity to handle internalization.
Luckily, Assistant could do the heavy lifting for you.
By integrating with Google Assistant,
your apps can leverage the powerful and evolving NLU
capabilities provided by Google.
Next, Georges will share more about
how your apps could integrate with Google Assistant.
GEORGES NABAA: Thanks, Luyan.
I'm Georges Nabaa, and I lead the Partner Engineering
Team working with Google's top partners on Assistant.
Let's look at how you can take advantage
of this infrastructure based on some common scenarios.
Depending on the type of app that you want to voice-ify,
Google Assistant can help.
If you're a hardware developer, check out our smart home SDK.
And if you want to create custom experiences,
look into conversational actions.
But for this session, we will focus
on how Android developers can take advantage of Google's NLU
infrastructure.
App Actions is the framework that
enables you to voice-ify your Android apps.
With App Actions, users say simple voice commands
to quickly access your app's functionality using
Google Assistant's intent mapping and NLU.
To implement App Actions, you need to do two things.
One, specify which user intents your app supports.
And two, specify how your app can fulfill these intents.
The first step is to let Google know
which user intents or capabilities your app
can fulfill.
To do this, you either leverage one of Google's Built-in
Intents, BII for short, or create your own custom intents.
Built-in Intents model some of the common ways
that users express tasks they're trying to do
or information they seek, such as ordering a meal,
booking a ride, or checking an account balance.
The great thing about BIIs is that Google
builds and maintains the language models required
to understand a wide variety of terms
that relate to that particular intent,
meaning you don't have to.
We now support 66 Built-in Intents across many verticals,
such as fitness, finance, food ordering, and more.
The second step is to let Google know how your app can
fulfill the user's intent.
This is done through an Android intent
that contains all of the extracted parameters
from the BII.
Now that we understand the basics
of intent and fulfillment, let's look at how we implement them
in practice.
In order to enable App Actions in your Android app,
you declare a capability tag in shortcuts.XML.
Capabilities tell Google what user intents your app supports.
Next, the intent tag contains the instructions
that tell Google how to build the Android intent that
specifies how your app can fulfill the user's query.
Intent tags also contain any parameters
that you want Assistant to extract from the query.
By default, Assistant sends BII parameters
extracted from the user query to your app
as extra data of the Android intent.
For example, for a user query like
"Hey Google, start my run on example app,"
the app receives an intent that invokes the component target
package and target class.
And the component receives an extra
with the key value pair Exercise and Run.
And here's how you access the extracted BII parameters.
Once you have a capability defined,
you can extend the functionality of that capability
with static shortcuts.
You can think of static shortcuts
as an instance of a capability with some of its parameters
having predefined values.
This is especially helpful if you
find yourself having to handle requests
that are variations on a theme.
For example, fulfilling the Start BII intent
requires your request matching logic
to handle variations like Jog, Sprint, or Race.
This logic quickly becomes cumbersome
as supported workouts increase.
You avoid this complex matching logic
by using an inline inventory, which
is a set of static Android shortcuts defined
in shortcuts.xml.
Each shortcut contains an item identifier
and a list of synonyms representing variations in how
users might refer to the item.
During invocation, the BII compares the user-provided BII
parameter to the synonym list.
When a match is found, the BII parameter
is updated to the matching shortcut's item identifier.
In this sample, the inline inventory shortcut
declares a parameter-binding tag within a capability-binding
element, binding it to the exercise.name BII parameter
defined in the capability.
And here's the string resource, at array run names, that
specifies a list of synonyms that Assistant will recognize
and map to the Cardio_Run item ID.
Note that static shortcuts are ingested by Assistant
when you upload a release to the Google Play console.
Since static shortcuts can only be created and updated
with new releases, they are most useful to highlight
common activities and content in your app.
Assistant performs the natural language understanding
on the query matching the semantics of the request
to the predefined pattern of a BII or custom
intent you define.
Then Assistant generates an Android intent
to launch the in-app destination of the request using
information you provided in the capability and shortcuts tags.
And finally, Assistant fulfills the user request
by launching the generated Android intent
and its resolved parameters.
At this IO, we're announcing a slew of new features
that further enhance the benefits of the Actions
platform.
First, Brandless queries now allow
Assistant to fulfill user requests through your app
even when your app is not explicitly named.
We are also announcing app install suggestions,
where Assistant can automatically direct users
to the Play Store listing to encourage them to install
your app if a user triggers the proper Assistant
query for the functionality of your app
when it's not installed.
With smarter custom intents, we've
improved the machine learning grammar expansion that
triggers your app, meaning you don't have to manually model
as many query patterns.
And finally, we are enabling shortcuts
to more Android surfaces like Auto and WearOS.
For more details, please watch the Assistant Across Devices
and Assistant on Auto talks to learn more.
With this, I will pass it to Rebecca to recap.
REBECCA NATHENSON: Thanks, Georges.
Google is working hard to make Assistant the best way
for users to get things done with their favorite apps
and devices.
We've seen that users want voice to work more seamlessly.
Google's investment in machine learning and NLU
can be leveraged by developers with App Actions.
And to learn more about how you can harness Google's NLU,
check out our other Assistant sessions
like Car Talk Assistant for Android Auto, which
goes into how Assistant supports Android app content on Android
Auto.
You can also learn about an overview of the capabilities
available to app developers across a huge variety of device
types at Google Assistant Functionality
Across Android Devices.
And finally, for hands-on learning,
check out our workshop, where you'll
learn how to integrate your Android widgets with Google
Assistant with App Actions.
Check out these additional resources to get started today.
And don't forget to follow us on Twitter
or sign up for The Developer newsletter.
On behalf of Alec, Luyan, and Georges, thank you.
[MUSIC PLAYING]