[MUSIC PLAYING]
JESSICA EARLEY-CHA: Welcome to "Assistant On Air,"
where we have conversations about building
for Google Assistant.
I'm Jessica.
And I'll be your host today.
I'm a Developer Relations Engineer for Google Assistant.
This episode is different from our typical two people
sitting on the couch and chatting.
We'll be hearing from Rebecca and Toni, who
work on Assistant, about the latest updates.
Over to Rebecca to kick this off.
REBECCA NATHENSON: Hi, everyone.
I'm Rebecca Nathenson.
I head up the Product Management team
for the Google Assistant Integrations and Developer
Platform.
With me today is Developer Relations Engineer Toni
Klopfenstein.
It always amazes me how deep the technology is
that powers Google Assistant.
For those of us who grew up with some flavor
of sci-fi voice-powered assistant
on TV or movies, something like the "Star Trek"
computer or Jarvis, it's easy to think
of a smart, helpful assistant as somehow within reach, at least
with a little movie magic.
But the truth is, in the real world,
we can't fix it in post-production.
We have to get it right from the start.
When I joined Assistant in 2020, I
was astonished at the depth of thought, design, and technology
that went into a simple utterance like, book
a table for four at 8:00, everything
from natural language understanding
to understanding which services can fulfill
the query to creating natural text-to-speech voices that
don't sound like a bad sci-fi robot.
And when we add in the ability to understand
what kinds of actions apps can take and let
the assistant connect users with those actions
wherever it makes the most sense, it feels like magic.
But that magic comes with a lot of very hard work backing it
up.
Let's talk a bit about where we are today.
We're still in the very beginning
phases of the helpful voice-assistant journey,
but we're seeing great growth.
The range of capabilities across technology is skyrocketing.
So it's no wonder that many of us
are adding more and more computing devices to our homes.
In 2019, the average US household
had 11 connected devices.
In 2021, that went up to 25.
I counted in my house.
Let's just say I'm pulling the average up a bit.
Now, with the proliferation of devices,
it's more important than ever that they
work better together so users don't
have to learn a bunch of new interfaces,
remember which capabilities work on which devices,
or pull up the same content or task over and over again
as they move from one device to the next.
Google Assistant is a key ingredient
that helps devices to work better together because it
provides a natural, intuitive interface for accelerating
a user's tasks across their apps and devices.
You can think of Assistant as the glue that links together
these devices with a consistent voice-forward user experience,
as well as letting users jump straight
to the most relevant or useful point in their apps.
Assistant does this by understanding what capabilities
apps support and then connecting that knowledge
with users' requests to deliver the best possible response
to a query.
This applies whether users already have that app installed
or not, whether they make the request through voice or touch,
and whether the request is a simple command
or a complex multi-step journey.
I'm also super excited about the advances
we've made across Google around better voice capabilities,
from building out conversational AI, like LaMDA--
a language model designed to engage
in multi-turn conversations on virtually any topic,
to crafting a more capable platform than ever
with multi-turn, smart dictation in Gmail.
Now, not every new capability makes its way
to developers on day one.
But developers can absolutely leverage the investment
we've made in Voice by integrating their Android
app with App Actions.
Now I'm going to pass it over to Toni
to take us through an overview of App Actions
and what it can do for you.
Take it away, Toni.
TONI KLOPFENSTEIN: Thanks, Rebecca.
I'm Toni, a Developer Relations Engineer for Assistant.
With App Actions, users can jump right
to the most interesting and useful points in your app
with simple voice commands.
When implementing App Actions, you
can think of it as having two main steps.
First, we need to process the user's spoken input
and understand their request.
Second, we need to fulfill the user's request.
To understand a user's request, Assistant has identified
and supports over 60 intents, which we call Built-in
In Intents, or BIIs for short.
These BIIs are also arranged by several categories.
Assistant does intent matching between the user's speech
to the BII.
Intent matching loosely means identifying several speech
patterns for a particular functionality
or intention of the speaker.
Let's walk through the user experience
and see who does what.
Here we have the user, assistant, and your app.
When the user says, hey, Google, order a pizza from example app,
Google Assistant will process the user's input
using Natural Language Understanding, or NLU
for short.
Assistant will do intent matching
and pull parameters supported by that intent.
The BII understands that the user is initiating an order.
And someone could ask for a specified menu item or cuisine
type.
This information is added as a field to the BII
so the app can provide exactly what the user wants.
Here, the order menu item BII is matched,
and pizza is pulled from the user's input
and set to menuItem.name.
An Android intent is sent with this information
to the app, which will open the app to the screen
for selecting pizza options.
Since Assistant takes care of all of the NLU,
all you need to do is select which
BIIs your app will support and how
you want to fulfill these user requests via Android intents.
It's never been easier to integrate your Android apps
with Assistant by taking advantage of Android shortcuts
and media APIs.
These enable fast, natural entry points to content and features,
not only in your mobile app, but also Android TV, Android
for cars, and Wear OS apps.
We're working to provide a consistent developer experience
across device types to ease the effort of voice enabling
your app wherever it runs.
We heard from our developers that it's
tricky to convey to users that there is a helpful new way
to access in-app functionality with their voice.
For example, if users asked Assistant for functionality
that your app can handle, but didn't say the name of your app
in their query or didn't have your app installed,
Assistant wasn't able to help them.
To improve this, we've launched more ways for Assistant
to get users to your app so you don't
need to worry about providing exact instructions
to your users on how to use your app with Assistant.
Two new features, brand list queries and app install
suggestions, are now available immediately for all App Actions
developers.
When a user queries Assistant without using a specific app's
brand name, also known as brandless query,
Assistant can now infer what app would best
fulfill the user request and effectively route the user
to that app.
Let's see an example of this with the Walmart app.
If a user attempts to check their shopping
cart without identifying Walmart by name in the query,
Assistant will still identify the desired functionality
and launch the Walmart app to the Shopping Cart page.
If a user triggers the proper Assistant
query for the functionality of your app,
even if the app isn't installed yet,
then Assistant can provide app install suggestions.
Assistant can automatically direct users to your Play Store
listing to encourage them to install the app and access
full functionality.
Let's see how this looks when a user doesn't have
the requested app installed.
If a user says, check my shopping cart on Walmart,
Assistant will open directly to the Walmart app Play Store
page, encouraging installation.
We've also improved custom intents as well.
For apps using custom intents, such as, schedule
an appointment, machine learning models can now automatically
increase the number of phrasings Assistant can recognize
that will trigger the app, meaning developers
don't have to come up with as many query
patterns on their own.
These features are automatically available to users
without any additional development work
required on your mobile app.
Let's take a look at how your Assistant
on-mobile implementations can be easily scaled
to different device surfaces.
We're excited to extend App Actions to Wear OS.
When you're going for a run or a bike ride,
chances are that you want to leave your phone in your pocket
and keep your hands free.
Your smartwatch is easily accessible,
but the small screen makes it hard to touch and type
or to digest a lot of information.
That's where Assistant comes in, enabling users
to issue voice commands to their Wear OS device
and get glanceable information in return.
The health and fitness BIIs used in your existing mobile apps
will be enabled on Wear OS devices,
meaning users can trigger the same expected integrations.
Let's take a look at how this will
work for a health and fitness app that has integrated
App Actions on Wear OS.
Users will be able to launch the Assistant,
say something like, start my run, to trigger the BII,
and Assistant will launch that app feature automatically.
Wearables aren't the only type of device
where users may want or need a voice-first experience.
Let's take a look at automotive applications.
If you have a use case for your app
that's also appropriate for use in cars,
App Actions works there as well.
Developers can add voice functionality to car apps
by expanding their mobile App Actions implementation
with the Car App Library.
Once done, these apps can be used on either Android Auto
or Android Automotive OS.
Android Auto enables users to connect their Android phones
to compatible vehicles and use their favorite apps
and services right on the car's display.
Android Automotive OS with Google built in
allows users to access Google Assistant, Google Maps,
and apps from Google Play directly from the car display
without the need for a phone.
The Car App Library provides templates
that can be used to build parking, charging,
and navigation apps.
These templates are designed with driver safety
in mind, minimizing distractions with
glanceable voice-forward UIs that
adapt to the vehicle they're running in.
At this time, App Actions on auto
supports BIIs that are relevant to on-the-go use cases,
finding parking and charging.
Let's take a look at how these BIIs
work with our partner ParkWhiz.
ParkWhiz allows users to find nearby parking spots on the go.
Here's a demo of the Android Automotive OS user experience
where the user says, hey Google, find parking nearby
on ParkWhiz.
App Actions are helpful for enabling voice commands
and apps that find parking locations, charge
stations, and more in the car.
But what if users want to listen to your media app?
If you've already built a media app,
Google Assistant can also be used
to bring voice-first functionality to your users.
When you implement a media session for your audio or video
app using the media session API, Assistant
can access the transport controls
for playback, allowing users to play, pause,
skip, or discover what's playing, all through a voice
query to their phone, tablet, automobile, or television.
Developers can add this functionality to existing media
sessions by setting flags for media and transport controls
inside their media session.
By using the existing media session transport controls
within your app, Android will display controls
for play, pause, seeking, or skipping
to an adjacent piece of content, depending
on which actions your media session supports.
There's so many ways to build a delightful experience
with Assistant on Android devices.
Back to you Rebecca.
REBECCA NATHENSON: Thanks so much, Toni.
We, at Google Assistant, are so excited to keep
building an Assistant that's the best way for users
to get things done with their favorite apps and devices.
We've seen that users want Voice to work more seamlessly,
and that's our goal every single day.
To learn more about how you can harness Google's capabilities,
check out our other Assistant sessions,
like "Assistant in Android for Cars,"
which goes into how Assistant supports Android apps in cars,
and "Assistant on Wear OS Devices,"
which covers ways to integrate Assistant into your Wear OS
apps.
Take a look at these additional resources
to help you get started today.
And don't forget, you can follow us on Twitter
or sign up for The Developer newsletter.
On behalf of Toni and me, thanks for joining us today.
JESSICA EARLEY-CHA: Thank you so much, Rebecca and Toni,
for sharing with us.
We hope that you liked today's show.
If you have suggestions of future episodes,
please leave a comment below.
Follow us on Twitter for updates on our newest episodes.
Don't forget to also sign up for our newsletter.
I've been Jessica.
Chat with you next time.
[MUSIC PLAYING]