[MUSIC PLAYING]
JESSICA JONG: Hello, everyone.
My name is Jessica Jong, and I'm a technical solutions engineer
on the Google Home team.
And together with Yeet Uyan, we'll
tell you how to make use of Analytics and tools
to ensure high-quality Google Home integration.
The goal of the session is to get
you familiar with the developer tools for Google Home
and to walk you through Google Home Analytics
in order to improve the quality of your Google Home
integration, monitor outages, and set up alerts.
This is a typical developer journey
that you will go through while developing your integration
with Google Home.
You can use our rich suite of development tools
to learn, develop, and debug, and use
Home Analytics to look for errors
and monitor outages even after launch.
Let's start with the Developer Tools.
Before you start building your own integration, you can,
without writing any code, simulate a Google Home
integration using the Google Home Playground.
This will help you understand how a smart home device works
in the Google Home ecosystem.
The Google Home Playground is a web application
where you can create virtual devices with the traits
that you like.
Let me show you how easy it is to create a virtual light.
First, you need to link Google Home
Playground to your account using the Google Home
app on your phone.
Now launch the playground in your browser and sign in.
Click on the Add a Device button.
Here, you can choose to add a custom device
or a predefined set of devices.
Let's create our own light here by selecting light type
and name it Super Light.
Note that the on/off trait is automatically
added to the device since it's a required trait for light.
We can add other traits after creating the device.
Now click the Create and Add Device button,
and the device is created.
On the Basic Information tab, you
can see the name, ID, and type of the device
that we just created.
Let's switch to the Traits tab to add a brightness trait
to the light.
Click on the Edit Traits button to add new traits.
From the list of Available traits,
select Brightness and click Save.
Now our light supports the trait of adjusting its brightness.
Let's switch to the States tab to view
the state of each trait.
You can manually adjust the brightness of the device
here or use a Google system to interact with the device.
Once you've gained some familiarity with the Google
Home ecosystem, you can begin your own integration
using the Google Home Extension for Visual Studio Code.
The Google Home extension lets you troubleshoot your Google
Home integration as you're developing it
in your favorite IDE.
The Google Chrome Extension provides all the tools
that you need for debugging in one place, like Google
Assistant Simulator, Cloud Logging, Home Graph Viewer Test
Suite, and other tools to simplify your development
process.
Let's take a look at this fantastic tool.
You can install the extension from our developer site
or directly from the Visual Studio Marketplace.
After signing in, select a project that you're working on.
Let's assume that you have completed a light integration.
From the Quick Access panel, click on Cloud Logging
to view the logs from the last seven days.
You can filter by Severity or by TimeRange.
You may also toggle the Real Time button
to view the logs in real time.
You can use the Google Assistant Simulator to send an utterance
to Google System.
And use Home Graph Viewer to check if the device
state has changed accordingly.
You can also send batch utterances
by running an utterance script.
Click on the Play button, and you
can see how it is sending the scripted utterances one by one.
The transcript can be edited and exported again for later use.
You might have noticed the Cloud Logging has
been showing real-time logs.
Switch back to the Home Graph Viewer
to view the updated states.
Now let's run some tests for the device.
Select the device that you want to test and click
on the Run Test button.
You will be redirected to the Test Suite, where
you can run tests for the device as many times as you want.
Whenever you're done, click on the Done button
to return to VS Code.
We'll be adding more features to the extension,
so don't miss out on the updates.
When you're ready to test your Google Home integration,
you can use the Google Home Test Suite to run end-to-end tests.
You might already be familiar with this tool,
as it is required to certify your Google Home integration.
We have been adding other features to the Test Suite,
like the Google Account Linking Validations
test for cloud integrations.
The validation test checks your all-service
for potential issues that may failed Account Linking
or token refreshing process.
We have also added tests to check
if there is any discrepancy in the device state return
by the QUERY intent and Report State.
This is how the Test Suite works.
It automatically generates test cases
based on the devices and traits in your HomeGraph.
The Test Suite will then simulate user engagement
with your Google Home integration.
The Test Suite will send utterances to Google Assistant.
This will result in executing intent being fired
to your cloud service or a Google Home device, which
will interact with the smart home device
and change its state.
Finally, your cloud service or Google Home device
should tell Google about the new state by sending a ReportState.
If the device state changes as expected--
that is if the resulting device state matches
with what the Test Suite expects to see,
the test is considered as pass.
Finally, when you complete the test,
you are ready to get certified and launch your Google Home
integration.
And now I'll hand it over to Yeet
to talk about how to leverage analytics to detect errors
during development and monitor outages after launch.
YIGIT UYAN: Thanks, Jessica, for giving us
an overview of the tools.
My name is Yeet, also technical solutions
engineer for Google Home.
Today, I'll be talking about Analytics.
Since last year, we have expanded Google Home analytics
into many areas.
We now have individual dashboards for each integration
type, including matter.
We wrote a new troubleshooting guide for smart home debugging,
created the confidence metric tracking your execution
success, and released a set of resources on detecting outages.
We are going to cover each of these throughout the talk.
But first, let's remember how our analytic systems work.
Each time when one of your users issues a voice intent,
Google will send a request to our fulfillment endpoint.
You take this request, execute on your devices,
and give us back the results.
The pipeline runs very similarly for local
and matter as well, which the main difference is
the execution happens locally.
So when you respond, if your result indicates an error,
it doesn't reach back to us.
Or if we cannot parse it, we count that as an error.
For this, we'll increment metrics and record an error log
on your project.
As the developer who wants to debug these issues,
the debugging flow would include two steps.
First, monitor these errors, specifically the breakdown
for error codes on a project.
And second, search these error records
in your logs to find more information about them.
Speaking of error codes, we create a list of error codes
that you can return to us in the Fulfillment Errors
And Exceptions page.
These include things like device stuck, door open, bin full,
low battery, and used to control the voice response,
Google Assistant gives back to the end users.
Then we have a set error codes that we
provide for cases such as the fulfillment response--
resent times out or returns back with no response.
We call these Platform Error Codes,
which you can find the full list in our troubleshooting page.
So noting these, let's cover a common scenario.
You are a prominent manufacturer of a smart light bulb,
and you work so hard to come up with a unique design that
works great at the same time.
People recognize your success, and your sale numbers
are increasing.
You made integrations with different ecosystems
like Google Home.
And leveraging them, you start to expand
into different regions around the world.
Congratulations, by the way.
You are on track to become a global brand.
So at the same time you invested lots of resources
to build an architecture that supports your global expansion,
now users around the world can use your devices
with very low latencies.
Your integrations are doing just great.
Then all of a sudden one day, you
wake up with a phone call learning
that the majority of your devices do not work anymore.
Your team is telling you the incident started
seven hours ago, affecting 85% of your devices,
and they are yet to figure out the root cause.
So I'll ask you, what do you do at this point?
And maybe more importantly, what can you
do to prevent something like this?
In this section, we are going to cover the steps to create
a pipeline to deal with the unexpected performance
drops on your project.
And the first thing we are going to cover
is monitoring the quality of your project.
With Google, the simplest way to do this
is through our dashboards.
You can access them in Google Cloud Console.
After selecting your project, go to Operations, Monitoring, then
Dashboards.
The dashboard we provide include the Google Home Analytics
[INAUDIBLE].
The first one you see is the main dashboard,
which gives you an aggregate view into your project
performance across all your installations.
You can find data for the number of requests your project's
handling, how the success rates of your integrations
compare to each other, as well as the daily, weekly,
and monthly active user numbers here.
The next set of three are the Integration Dashboards,
which gives you a detailed view into how each
of your integrations are doing.
They contain data for reliability,
usage patterns broken by device types and traits,
list of top error codes, as well as
any metrics unique to that integration type.
You'll only have data in these dashboards for the integration
types you implemented, whether it's cloud, local, or matter.
And we just covered using our metrics and dashboards
for monitoring quality.
Now it's time to take a look at identifying outages.
We have created a new Codelab for monitoring outages
for Smart Home.
It's accessible through developers.home.
google.com/codelabs.
And I'll try to give some highlights of what's
mentioned over there.
First of all, it's valuable to watch out for three key outage
indicators on your project.
During an outage, you'll see a sharp drop in your success
percentage chart.
Usually when that happens, it will
be accompanied by an increase in your request latency,
as well as a change in the error breakdown chart.
When this happens, the first best move generally
is to take a look at your error breakdowns
to see which errors might be contributing to your outage.
Another outage indicator we created this year
is the execution success confidence metric.
By comparing the current execution performance
with your historic averages, this metric
assigns your score between 0 and 1 every 15 minutes.
1 means your project is performing
at or above the historic trends.
And 0 indicates that we have detected
a sharp drop in your execution success,
likely indicating an outage.
This metric is based on standard deviations.
So if your project has been historically very, very stable,
a sharp 20% drop in your execution success
might be enough to get this number down to 0.
When identifying outages, we recommend
to watch out for cases where this metric drops
below 0.5 very closely.
Now that we know how to identify the outages,
it is time to set up the alerts.
Automated Alerting is a feature of Google Cloud that
allows you to watch out for certain changes on your metrics
automatically.
The policies for alerts are set through the Alerting view,
which you can access through the Google Cloud Console
menu, selecting Operations, Monitoring, then Alerts.
Once there, click on the button that says Create Policy.
Creating an alerting policy window
will walk you through three steps.
First, you need to indicate what you are going
to track by adding a condition.
This should bring up a view that you
can select a metric and configure conditions.
For selecting a metric, you can type "smarthome"
and see the list of metrics that we provide on your project.
To monitor outages, select the Execution success confidence
from the list.
Then as a condition, you can configure it in a way
that it triggers if the time series is below 0.5
for the most recent data point.
Then the second step is to configure
a few settings that essentially control who will be notified
and how.
Notification Channels is the list
of channels to receive the alerts.
We'll use the email as a channel option.
And then through our email here, you
can also receive alerts via SMS, pop-ups,
or through third-party integrations
if you choose to configure so.
There are also fields to specify the incident closure behavior.
Notify on incident closure sends a separate email
when the incident is closed, and the incident autoclose duration
specifies a period to automatically close
the incidents if no action is taken.
And the final step is to give your alert a name
and write a short documentation.
The documentation will be sent to people with your alerts.
So make sure to include the steps to troubleshoot.
For example, you can include the link
to our troubleshooting guide.
Once you save your alert, it's going
to be listed in the Policies section of the Alerting view.
There is also an incident section
to see the past events triggering your alert.
From this point on, hopefully, you don't get any alerts.
But if you do, you'll receive a notification in email
indicating when the incident happened
and what's the observed value.
In the example, we set our threshold to 0.5.
And the observed value came out as 0.35,
which is below our threshold.
So when you get this alert, it is
time to go back to the Debugging Flow,
and take a look at your error metrics,
and search them in your logs.
In this talk, we covered the Developer Tools and Analytics
for our smart home ecosystem.
You learned how you can use them to have
a high quality and reliable Google Home integration.
My name is Yeet.
My colleague was Jessica, and we hope
what we have seen here today helps you
with your Google Home integration.
Thanks for joining us.
[MUSIC PLAYING]