Hi, everyone. My name is Yiǧit.
I’ll be presenting our debugging tools today.
Smarthome Analytics is a set of tools available
on Google Cloud Platform to monitor your project metrics
and analyze your logs.
And Test Suite is an online tool
that creates an automated test plan for your devices
and executes it against your smart home integration.
Both of these tools are available today.
You can use them with the cloud integration
for smart home, as well as with Matter projects.
If we open up a quick note on Matter,
our tools will give the ability to monitor outages
and quality issues in production, insights into errors,
and with that, the ability to troubleshoot issues in a self-serve manner,
and a streamlined way to test your implementation
to validate your changes and updates.
Before we go a little further, let’s review a few key items.
During this talk, you’ll be hearing a lot about metrics and logs.
Metrics are counters that represent the user interactions with your devices
and provide breakdowns on various components.
And logs are JSON-like strings that contain the details for those interactions,
which is especially useful for things like storing an error message.
I did put a high-level example for both on this slide.
You see the metric contains the aggregate data,
and the log provides details like device type, trait, and error message.
So, essentially, if you think the metrics can help you
to determine where your action is having trouble,
and logs are useful when making a deep dive to solve these problems.
Our goal is to create a debugging loop using these tools
where you start by checking the metrics, see the gap in your success rate,
identify your top error, and search this in your logs
to see the error description.
And, hopefully, you will be able to utilize that info to fix the issue.
There are lots of metrics we provide to a project, by the way,
such as the success percentage, 95th percentile latencies,
daily active users, error breakdowns, and so on.
So, knowing these, we’ll go through our “Debugging the Smart Home” codelab.
This is a learning resource for you to try out the basics of our tools.
It is accessible following this link,
and I took some screenshots so we can walk through it together.
You can complete this codelab with the virtual washer provided there,
or you can use a Matter project.
For the virtual device option, the codelab provides a virtual washer,
which we added two errors intentionally.
It has a section to walk you through the steps
to get your project up and running in a few minutes.
And to bring a Matter project instead,
you can attend Don’s workshop “Embedded Onboarding Preview.”
In the workshop, he walks you through the basics of setting up a device
that works with Matter.
So we’ve got our device. We are ready to start testing it.
For that, I pulled out my Google Assistant on my phone
and issued a few test phrases like, “Hey, G, turn on my washer,”
or “Hey, G, pause my washer.”
I do have a short recording for you in the next slide.
So let’s see what the results will be.
I’ve recorded the conversation in writing on the right.
So the first call is “Okay, G, turn on my washer,”
and the Assistant responded with “I got it. Turning on the device name.”
But when I said, “Okay, G, pause my washer,”
the response came out as “Sorry, I couldn’t reach the action name.”
As you see, the first one went through, but there’s an error with the second one.
So what’s going on behind the scenes?
Each time one of your users issues a voice intent,
Google will send the request to our fulfillment endpoint.
You take this request, execute it on your devices,
and give us back the result.
If your result specifies an error, or if we cannot parse it,
we will count that as an error.
And slightly different to cloud execution, with Matter,
Google will deliver the intent right to your device.
If there’s an issue with the execution flow,
we’ll also count that as an error.
For either of these cases,
we’ll increment the metrics and record an error log on our project.
Your debugging flow would include two steps.
When an error happens, monitor these on your project metrics,
then search for the error codes in your logs.
Both of these are available to you right out of the box.
Okay, so we’ll start at step number one. That is monitoring your error logs.
And, for this, we’ll go to Google Cloud Platform,
choose our project, then clicking the menu icon to bring up the side menu.
Then we’ll scroll down to Operations, and from there, Monitoring, Dashboards
is where you can find the metrics.
And Logging, Logs Explorer is where you can find your logs.
Okay, we are starting with the metrics, so we’ll select the dashboards.
This will take you to your dashboards page,
and by default, you’ll see that there are extra dashboards
that Google Cloud makes available to your project.
We are publishing all our data in dashboards named Smarthome Analytics
so that will be the one to access your project metrics.
You’ll see that we do have different charts for each of our metrics.
Starting with the upper left corner,
Smarthome Requests shows the number of requests we issued to our project.
It shows two in this case for the requests that we just issued,
and we have the success percentage chart in the upper right corner,
which shows as 50%.
This is because one of our requests have succeeded,
and the other one has failed.
And we do have the breakdowns for execution and query,
which shows us the counts for successful requests.
Our query chart is empty because we haven’t issued any queries.
That would be something like, “Okay, G, is my washer turned on?”
but we have only issued executions,
which is something like, “Okay, G, pause my washer.”
Then if we scroll down, you’ll see some other charts as well.
You’ll see that we have one for latency.
We use the 95th percentile as a golden metric for latency within Google.
This one shows around 900 milliseconds.
Then the Active Users chart is another one we have.
That chart will have three data sets:
one for daily, one for weekly, and one for monthly active users.
The data in our charts show up pretty much instantaneously.
It should not be more than a few minutes max.
The data in the Active Users chart, however, is an exception to this.
We are publishing aggregate data there once a day at midnight,
so you'll see the usage corresponding to today
when you check this chart tomorrow.
That’s why you are seeing it empty.
Then, the last metric that I want to show you is the error breakdown.
This will be crucial to debug errors, as it gives a breakdown of our error codes.
The error we have there shows “PARTNER_RESPONSE_MISSING_DEVICE.”
From the charts we have seen in our dashboard,
I think there are three key ones that can provide great insights
for monitoring quality and debugging.
Success % shows the overall success percentage
of all smart home fulfillments.
Request Latency shows the 95th percentile of perceived latency on Google’s servers.
Different devices does have different requirements for these, by the way,
so I recommend checking
the specific guidelines from our documentation.
For a washer, as we are covering in this talk,
the success percentage should be about 97%,
and the latency should be below 3,000 milliseconds.
And also we have the Error Breakdown chart, which is crucial for monitoring your errors.
Then we'll continue with step number two, which is searching through your logs.
The error code we saw in the previous step was "PARTNER_RESPONSE_MISSING_DEVICE."
You’ve got to remember that one.
For this, we’ll go through our Logs Explorer.
This tool does have two main sections.
A query field to write queries to search through your logs, the one on top,
and a query results window that shows the logs matching your queries.
In the query field, I’ll write “PARTNER_RESPONSE_MISSING_DEVICE” in quotes,
then run my query.
As you see, it was able to identify
the error log matching this execution flow.
You can click on it to expand the fields.
As you see, the log provides some useful information
like what was the device type,
the device trait associated with this execution,
what’s the latency,
as well as the request ID that you can follow up
in your own server as well.
Each of our error logs, you’ll see a status field with a status type,
which matches our error code and an external debug string,
which contains the error description.
This one says, “JSON response does not include the device,”
so there might have been something wrong when we were preparing our JSON response.
You saw an example conducting a string match as a filter.
Similar to this, we can also fetch certain fields within the log,
like isSuccess = false and deviceType = “LIGHT”
which will give us the error codes coming from lights.
As you see, we can combine these prompts with Boolean operators,
such as AND, OR, NOT, and numeric fields. You can use comparisons.
Latency millisecond greater than 300 will get you the logs
that have higher latency numbers, for example.
Google does provide a query library, by the way,
that has many different examples.
You can see the link below to follow up on that.
Moving forward, let’s turn to our code to do some debugging.
We have mentioned that our functional code is stored in the functions directory.
In there, you’ll find an index.js file that has all our implementations
for the smart home intent fulfillments.
Our smart home ecosystem defines three main intents:
sync, query, and execute.
You’ll recall that we noticed our error when we issued
the, “Okay, G, pause my washer” command.
This is an execution intent, so we are going to start our debugging
through our execution function at line 210.
There’s a nested loop sequence that goes through
the request fields and calls the “updateDevice.”
And there isn’t much else in this function,
so if there’s an error, it has to be with the update device function.
Taking a look at the “updateDevice,” we’ll see that it’s an asynchronous function,
a few variable definitions,
then you’ll see a beefy switch that compares the command value to different cases.
Each case represents a trait here, and if you look closely,
the PauseUnpause trait has an extra Pause at the end.
So we are deleting that extra pause.
This is actually a very common error that we have seen across the developers,
so always double check your trait names.
Once the issue is fixed, we’ll do another Firebase deploy
and test our action.
As you have seen, the pause command failing with
“Sorry, I couldn’t reach the action name” before
comes out as “Sure, pausing the device now.”
So congratulations! We fixed our first error.
The way that we have tested our action was through issuing a few commands
and checking the behavior log.
This is an easy way to test the general behavior
maybe on a single device,
but not so much to test your whole implementation.
For a more comprehensive test, we will use Test Suite.
Okay, we have advanced to the fourth section of our codelab now,
which is to test your action with Test Suite.
This page of the codelab will give you the steps to run the Test Suite,
but I'll give you a quick intro instead.
Test Suite is our test platform for smart home actions.
This tool requires a project ID to identify your project
and a signed-in Google account with devices to test.
It checks which devices are available on that account
linked to this project, generates an automated test plan,
then runs these tests by issuing commands and checking the state changes.
The tool overall provides regression testing capability
through your entire application life cycle.
This is a tool that we recommend you to run periodically
to catch any errors before you're releasing to production.
The Test Suite is available through the link I provided below.
I did also grab a quick video, running Test Suite on my project,
which we can run to see how the tests go.
So, first step, I’m signed in with my Google account,
then my project ID, I’m entering it into the web application.
The tool was successfully identifying my device,
and it provides some extra settings for tests as well.
I’m turning all of them off for now, then clicking Next.
For every trait that it found on my device, it’s going to do a series of tests
for Boolean traits like OnOff.
It’s going to do one test for the “on” case,
one test for the “off” case, similar to the “StartStop.”
Then, for each of these subcases, it’s going to do basically one execution,
grab the execution result.
Then it’s going to expect to receive a report state,
then grab the report state result.
Then it’s going to issue a query.
And comparing the report state response with the query response,
it’s going to do one set of tests to check the accuracy of that as well.
So, as you’ve seen, the tests still go.
I did grab a screenshot of the final version of all the tests.
So, fast forwarding to that one, we’ll get this.
Okay, so as you have seen,
there is still an error with the “pause the washer,”
and we see that it says: expected state to include “isPaused = true,”
but the actual state does not contain the “isPaused” result.
As a result, both of these execution report state pairs,
as well as the discrepancy check for report state and query, fails.
From our earlier tests, we know that our execution works now.
So this issue has to be on the report state side.
Our report state implementation is at line 277.
You’ll see that we are building a request response
by including a request ID, agent user ID, device states, and so on.
A careful look at the list states.
You’ll realize that we have the “isPaused” missing here.
So we’ll add the, “isPaused” like this and rerun Test Suite.
And as we see this time, all our tests pass,
and we can confidently say that we have identified both our errors
and fixed them using our tools.
As a recap for the whole workshop, we learned how to monitor errors
by using the error breakdown and recording the error code.
We then used the Logs Explorer to filter through our logs
and found the error log matching our error code.
Seeing the error description helped us to fix our first bug.
We have also covered using Test Suite, our regression testing tool.
As I mentioned then, I do recommend using this tool
whenever you are testing for a bug, testing after a change or even periodically
to ensure your smart home integration works smoothly.
Well, thank you for staying with me to the end.
Besides the tools that I showed today,
we are also working on providing documentation and tooling
to create alerting systems on your projects.
Stay tuned for more updates on these in the upcoming weeks.
Again, my name is Yiǧit.
Hope what I’ve covered today helps you to debug your smart home projects.
Thank you for watching.