[MUSIC PLAYING]
JESSICA DENE EARLEY-CHA: Welcome to Developing for Assistant
across Devices.
I'm Jessica, and I'm a Developer Relations Engineer for Google.
TONI KLOPFENSTEIN: Hi.
I'm Toni, and I'm also a Developer Relations Engineer
for Google.
JESSICA DENE EARLEY-CHA: Android apps are already
helpful every day for users.
Integrating your app with Google Assistant through App Actions
can extend this helpfulness even further.
App Actions allows users to launch and control Android apps
with their voice.
So they could say something like, hey, Google,
order a pizza from example app to Assistant,
and the Android app will not only open the Android app,
but also to the section of the app
that they could start their order.
We've brought the Assistant and Android closer together
to make it easier for developers to quickly develop and test
voice integrations for their app.
By implementing App Actions, you could streamline user journeys,
where a user can share key information using their voice
and be dropped into the correct section of the app,
along with any parameters provided.
This also helps eliminate the issue
of searching through the dreaded sea of icons.
When a user queries Assistant without using the brand
name of a specific app, Assistant
can now infer what app would best
fulfill the user's requests and effectively route the user
to that app.
This is called brandless query.
By implementing App Actions, Assistant
can service your app content to users.
If a user triggers the proper Assistant
query for the functionality of your app,
even if your app isn't installed yet,
Assistant can provide app-install suggestions.
Assistant can automatically direct users to your Play Store
listing to encourage them to install the app and access
full functionality.
Dynamic shortcuts let users jump into personalized place
or bit of content in your app, based on prior actions.
These shortcuts are created at runtime
and can be personalized to the user.
Also, during contextually-relevant times,
Assistant can proactively suggest your Android
dynamic shortcuts to users, displaying it
on Assistant-enabled devices.
For example, if you're building a note-taking app,
you might want to push dynamic shortcuts
based on the names given of the notes that your users create.
For the most critical user journeys,
you can suggest the user to create an Assistant shortcut
in your app, which is a phrase that
triggers a particular shortcut.
This is similar to how Assistant shortcuts can be created
manually by your user, but the in-app promo SDK
enables you, the developer, to proactively
suggest and implement Assistant shortcuts to your users.
For example, at the end of purchasing our pizza
from the example before, you can offer the user
to create an Assistant shortcut.
So next time, they could say, hey, Google,
get my usual pizza.
The Android app opens with the user's preferred pizza
information, and it's ready to be ordered.
And that promotion allows developers
to initiate the creation of an Assistant shortcut
by constructing a special deep link that
has the app's own deep link parameters inside it.
TONI KLOPFENSTEIN: Let's take a closer look
at how these user journeys are handled
at a high level between the Android app and Google
Assistant.
You can think about this experience
as having two parts--
first, processing a user's input and understanding
their request and, second, fulfilling the user's request.
When the user says, order a pizza from example app,
Google Assistant will process the user's input
using Natural Language Understanding, or NLU
for short.
Assistant will figure out what the user wants to do.
Then, the app is triggered with the user-defined data
and opens to the appropriate app feature to start a pizza order.
With that understanding of the high-level architecture,
let's dive in and see the differences
when developing for voice across different device types,
like mobile, wear, and Android for Cars.
Regardless of your device type that you're developing for,
you'll have to build, test, and release your app.
We have three different example apps
we'll be looking at as we go--
a fitness app for Wear, a to-do app for mobile, and a parking
app for Cars.
JESSICA DENE EARLEY-CHA: The first step
in building for voice is identifying
your app's functionality and expressing that to Assistant.
Think about what your app does, what type of functionality
it supports.
Do you have a game app?
Does it have a leadership board?
Does your app track workouts?
How about searching and viewing news updates?
A key concept in machine learning and NLU
is intent matching.
Intent matching loosely means identifying several pieces
of speech pattern for a particular functionality
or intention of the user.
For our example apps, we'll identify
these CUJs and intentions of each device category.
Assistant has identified and supports
over 60 different intents, which we call Built-In
Intents, or BIIs for short.
These BIIs are also arranged into several categories.
Google builds and maintains a language models for BIIs
to understand many common variations of queries related
to that particular intent.
Currently, for Wear, we support the health and fitness BIIs.
While for Auto, we support transportation BIIs,
like GET_PARKING_FACILITY and GET_CHARGING_STATION.
We can then combine our identified app capabilities
with the supported BIIs to start constructing voice capabilities
for our sample apps.
For Wear, we can use the start and stop exercise BIIs.
For mobile, we can use the OPEN_APP_FEATURE and GET_THING,
which are from our common by categories.
OPEN_APP_FEATURE allows users to request
which list they would like to see,
whether it's the completed or active,
and GET_THING uses the app search
to locate a particular item.
Now, for, Cars we'll use the GET_PARKING_FACILITY BII.
Some BIIs use inline inventories.
Inline inventories function like a lookup table
for BII parameters, expressing the variety of ways users
refer to a feature or content in your app
into items' identifiers that you define.
You'll need to define the exact set of words in the inventory
that you want Assistant look up for your users' requests.
These options are declared statically
in your app's configuration.
A unique identifier for the option
is received at runtime, as a parameter of the incoming BII.
Let's walk through how this is done.
Here, we have a user saying, hey, Google,
open ToDo and show my active tasks.
Assistant will match the OPEN_APP_FEATURE BII
and extract "active tasks" as feature.
Feature is the parameter supported
by the OPEN_APP_FEATURE BII.
In your inline inventory, you'll want
to look up the value active task in your app's configuration
and return the unique identifier of active.
Finally, triggering your Android app
with the OPEN_APP_FEATURE BII capability
to get the fulfillment with a parameter
feature value of active.
Looking again at our example apps,
we could start using the inline inventory
to clearly identify features in our apps for our users.
For example, apps on Wear will have the inline inventory
item of exercise name.
In this example, we're using "run."
As shown for our mobile app, we'll
have "active" and "completed" as our feature inventory items.
Well, for our Auto apps, we don't need
to use an inline inventory.
Capabilities are the expression of a relevant feature of an app
and contains the built-in intent and fulfillment instructions.
Capabilities live in your shortcuts.xml file.
Here, we have a capability that is
tethered to the OPEN_APP_FEATURE BII,
and our inventory parameter is called "feature."
To list out what possible values for feature,
like active and complete, we'll need to add a shortcut.
Within the shortcuts tag, you'll need
to add a shortcut for each type of feature.
Here, we have the shortcut for the active feature.
This includes a capability-binding tag
to the OPEN_APP_FEATURE BII.
This tethers this shortcut to the OPEN_APP_FEATURE capability
declared previously.
Finally, we'll have the parameter-binding tag
to our array resources that list out the synonyms for the term
active.
Here's an example of an array resource
with a list of synonyms for the term "active."
It's important to try to capture all the possible synonyms,
since Assistant treats this as a lookup table.
To capture "complete" as its own feature,
you'll need to add another shortcut with the complete task
information.
Here's a reduced version of the capability and two shortcuts.
To have your app know about these capabilities
and shortcuts, you'll need to add a metadata
tag that points to where your shortcuts files is located,
and make sure that it's within your activity.
There are a few differences when it
comes to developing for Wear and Cars to keep in mind.
Apps may require additional error notification
handling to users, depending on the standalone status.
Current exercise states are not sent
to Assistant along with the health and fitness BIIs.
So if your app is not declared standalone,
you'll need to build an additional haptic feedback
to inform users whether their request was successful or not.
While developing apps for Cars, you'll
need to clear the our Car App Supports deep links.
You can do this by creating an intent filter for deep links
in the Car App activity.
TONI KLOPFENSTEIN: The next step is
to tell the Assistant how to fulfill
the BII that was matched.
Now that we have the BII and the in-line inventory for features
in place, we'll need to add our intent tag to the capability.
The intent tag contains the instruction
on how to build the inter intent that
will be delivered to the app in order to launch it.
The parameter tag actually lives inside the intent tag.
Note that the naming of parameters between capabilities
and shortcuts are important.
The capability's Android name has
to match the parameter bindings Android key.
Here, we're instructing Assistant
to take the parameter named feature from the BII,
and copy it into the intent extra named feature param.
In this example, the to-do app uses the value
of OPEN_APP_FEATURES feature param,
to filter the list of to-do items by active.
Let's take a look at the fulfillment differences
for Wear and Cars.
There are several locales supported for the fitness
BIIs on Wear, so developers will need
to verify their apps fulfill user requests properly
for each supported locale during testing.
Android for Cars only supports deep links,
not Android intents.
When handling the deep link, we specify
the fulfillment implementation in our Car session.
If a session is not created, an intent to start the app will
create the session and pass the intent in the onCreateScreen()
method.
If a session is already created in memory,
the session will reuse the same intent and pass the intent
to the onNewIntent() method.
Here, our Auto app gets the value
of get charging station deep link when creating a session.
The deep link contains all the parameters matched by the BII,
so the app can act on what the user said.
Similarly, when a session is already created,
the deep link will be passed into onNewIntent
with all the parameters matched BII.
Here is an example of how an Android
app gets the value of Get Charging Station deep link.
JESSICA DENE EARLEY-CHA: When it's time to test your app,
you can use the Google system plugin through Android Studio
to verify your app action integrations are configured
correctly.
First, you'll need to install the Google system plugin
from the plugin marketplace.
After the plugin is installed, you
can use the App Actions test tool.
With the emulator and the app installed,
open the App Actions test tool and then create a preview.
Then, you can select the BII you want to test.
For our ToDo app, we'll do the OPEN_APP_FEATURE
and change the feature value to active.
Then, run App Actions, which will open the ToDo
app to active tasks.
Normally, Assistant gets a hold of your app's App Actions
configuration at the time where the app's APK is
uploaded to Play.
It extracts the resources from the bundle
and readies itself on the backend to listen for the BIIs
from the configuration.
The test tool preview is a temporary backend configuration
that tells Assistant for the App Actions configuration
to be used for the specific app for a specific Google user
account.
This means, after the initial upload that
proves the developer owns the app,
the app doesn't need to be uploaded to play
each time testing is required.
You can use the test tool to preview your App Action
configuration for Wear devices, as well as on Auto.
TONI KLOPFENSTEIN: Now that you've
tested that your App Actions integration works,
you can request a review and deploy.
Before submitting your newly uploaded Android app
to the Google Play console, you'll
need to accept the Actions on Google Terms of Service
under Pricing and Distribution.
The App Actions review does not affect your Android app review
and deployment status in Google Play.
After the app is approved, the App Actions integrations
will be available to users of your published or open test
releases.
We've covered a lot of details on how to add voice
capabilities to your apps.
Let's recap briefly.
Giving users the ability to engage with your app
through voice requires the use of built-in intents,
the addition of your app-specific capabilities
to shortcuts.xml, how the Assistant should handle
fulfilling user queries, and this can all
be tested quickly with the Google Assistant plugin.
Assistant can also automatically leverage your App Actions
to help guide users to your app through fulfillment
of brandless queries and app install and suggestions, all
without any additional development work on your part.
And when developing for apps targeted
at devices other than mobile, there
are just a few differences to remember.
If you're ready to start adding voice to your Android apps,
check out the developer docs and code labs to get started.
On behalf of Jessica and myself, thanks for watching.
[MUSIC PLAYING]