[MUSIC PLAYING]
FERGUS HURLEY: Hey, everyone.
Thanks very much for joining today.
My name is Fergus Hurley, and I'm the product manager
for Android Vitals.
I'm really excited to talk to you today
about the past, present, and future of Android Vitals.
To start off, what is Android Vitals?
I'm sure a lot of you guys are aware of it already,
but just to make sure, Android Vitals
is Google's initiative to help improve
the stability and performance of Android devices.
We're doing a lot of different things
to be able to improve this.
One big part of it is within the Play Console,
we now have a section called Android Vitals.
And that's where we report to you different performance
metrics we're going to cover in this talk.
"How do we actually power this product?"
is an interesting question to start off with.
Well, hundreds of millions of users
have opted into sharing their device usage and diagnostics
data with Google and you, our partners.
And so we share that data in a privacy-compliant way
within the product.
I'm going to start by going through some lessons we've
learned over the last 18 months since we launched Vitals at I/O
last year from top developers out there that
are big and small, that have actually used Android Vitals
and got a lot of success from using it.
So hands up here.
Who wants less one star ratings?
If you don't have your hand up, you
might be at the wrong conference, unfortunately.
But most people want to avoid them.
And what we found when we looked at the review corpus
on Google Play, which is pretty large,
is that 40% of all one star reviews
where we extract a topic are talking
about stability and bugs.
And over the past year, the percentage
of users talking about stability and bugs in the reviews
has gone down by 18%, which is pretty amazing.
So we're going to talk about some
of the developers who have actually
contributed to this reduction.
The "Merge Dragons" team, which is behind a very popular Zynga
game, they used Android Vitals to be
able to pinpoint where specifically there
was an issue with their application.
And this resulted in them being able to reduce their one star
reviews by 50%.
The very popular Reddit app used Android Vitals
to be able to reduce their crash rate by 75%.
And they did this because they were
able to see the category benchmarks in the Play Console
Android Vitals section.
And that was able to make the case with their leadership
that they had room for improvement.
And then when they actually improved
their metric, they were able to show leadership, hey,
we've actually done what we set out to do.
And now we're one of the most stable apps out there.
The LOVOO team signed up for the Anomaly Alert emails
in the Play Console.
And as a result, they got notified
as soon as there was an issue with the ANR
rate of their application.
They had a huge spike caused by an SDK
that they were using in their application.
And even though you might be building an amazing product
yourself and think that it's very stable,
I'm sure you've used third party libraries.
And some of those could introduce issues
in your application.
So you want to become aware of these things
as soon as possible.
And the Anomaly Alert emails help you with that a lot.
They quickly disabled the ads SDK that was causing the issue.
And then they shipped a new fix that actually reduced their ANR
rate back down to 0.25%.
The ABA English team is a pretty small team.
And they don't have infinite resources to be
able to spend on performance.
And we don't think that you should spend all your time
focusing on performance either.
But we realize that you have to keep an eye on performance all
the time, and that it might make sense for you
to take a quarter to actually just focus
on improving the performance of your application
and hold off on new features.
Maybe do that as part of your 2019 Q1 planning
or do a spring clean for Q2 of next year.
The ABA English team did this.
And they spent one quarter, focused effort, and reduced
their ANR rate by 97% as a result. This and a bunch
of other things resulted in them being to increase their star
rating to 4.6 stars.
Now, obviously, you can't get a 4.6 star rating
but just focusing on vitals, but we
guarantee that you'll never get such a high rating
if you have terrible vitals.
So I've gone through some examples of individual app
developers who are using vitals.
What we've actually also done is we looked at of the top 1,000
apps on Play, we looked at their engagement in the Vitals
section since I/O. What we found is
that the developers who engaged most in the Vitals section,
like the top 10%, they reduced their crash rate by 50%
over that time period.
And this isn't just good for those developers
and those apps.
This is really good for Android N users.
This resulted in 3 billion more stable installs
on the platform.
We've talked about stability.
Another major area that users talk
about when they leave one star reviews
is resource usage issues.
And this is battery, network, memory, et cetera.
And we have many metrics in the Vitals section around battery.
And some of these metrics are interrelated.
So one of the metrics we have is excessive network usage
in the background.
And so this is bad for users in two ways.
It's consuming their data plan, and it's wasting their battery
in the background.
We have two metrics that we especially
focus on in this area.
They are wakeups and wakelocks.
I'm going to come back to them in a bit.
But the Jaumo team focused on their wakeups rate.
And what they say is that they wouldn't
have even noticed or been able to fix their excessive wakeups
issue without using Android Vitals, because we have
a lot of insights into your battery usage
that there are very few other tools that can actually
help you with.
This resulted in them being able to reduce
their excessive wakeup rate by 70%.
Great.
Now we're going to jump over to how you can actually increase
your five star reviews.
And what we find is that over the past year,
we've had more users mentioning speed, design, and usability,
which is what 70% of the five star reviews
talk about when we are able to find topics in them.
The MercadoLibre team, which is one of the largest e-commerce
apps in the world--
it's number one in Latin America--
they were building a new feature into their application.
It required them to ask for a specific permission.
This resulted in a bad user experience
where users actually had very high denial
rate of this permission.
There were able to see that in Android Vitals,
and then be able to redesign that feature to give users
an explanation of why they needed it, that permission,
and only ask when the users actually really
needed that for that specific feature,
and not really early on in the process.
The MercadoLibre team is a pretty big team.
And one of the things they found very useful with Vitals
is the ability to be able to keep track
on many of the metrics and be able to see
that at their startup time there was a regression introduced,
which caused the startup time of their application
to become very long.
And so they were able to then figure out
which team specifically had caused that issue,
and then work to be able to debug that issue.
The LOVOO team used the Android Vitals startup
time and the Firebase Performance Monitoring tool
to be able to really get deep insights into the startup
of their application.
So the Android Vitals covers the platform-level data collection,
but doesn't cover exactly in your application
where the issues might occur.
And so that's why we encourage you to use Firebase Performance
Monitoring as well to be able to understand where
specific issues are occurring.
They were able to figure out that their login
and sign up flow was taking way too long.
And they decide to do a rewrite of that part
of the application.
So what's in Vitals today?
So Vitals consists of five performance areas,
which I think you'll all agree are
things that you don't like to have
happen when you're using apps.
You don't want them to crash.
You don't want them to drain all your battery.
You don't want them to be slow in rendering or requesting
permissions they don't need, or be slow to start.
And so we have 15 metrics covering these five performance
areas today.
And across each of these metrics,
we provide three dimensional breakdown at a minimum.
So we show you OS version, your APK breakdown, device
breakdown, and where relevant, we provide other information
as well when it's possible.
A new feature that we launched at I/O this year
is category benchmarks.
This is where you can now be able to see,
compared to other apps in your category,
how do you stack up relatively so to your peers
across every single metric?
And we give you the 25th, 75th, and even
50th percentile breakdown for each of those category
benchmarks.
And that can really help you to be able
understand, OK, this metric I'm doing OK on, and this one,
I'm actually falling way behind.
I need to invest there.
I mentioned the Anomaly Alert emails earlier.
These are available in the console.
You have to opt in to receive them.
And so I encourage you if you're not
opted in already to sign up today and encourage
the rest of your team to sign up as well.
This where we notify you when there's
a significant change in the ANR or crash
rate of your application where we have a spike in the clusters
or across the core vitals, which I'm
going to talk about right now.
There are 15 Android Vitals metrics.
It's a lot of metrics.
We think all of them are important to deliver the best
possible user experience, but some
are more important than others.
As talked about earlier, the leading contributions
to one star reviews are stability issues and battery
issues.
So we have four core vitals that are divided between those two
areas.
So crash rate and ANR rate--
ANR rate being Application Not Responding--
are the two stability ones.
And stuck partial wakelocks and excessive
wakeups being the two in the battery area.
For each core vital, we provide you a bad behavior threshold.
This is where if you're above that threshold,
you're sort of failing the class.
That bad behavior threshold is established
by looking at the top 1,000 apps and seeing,
the bottom 25% of apps are above this rate,
and so that's where the bad behavior is.
We launched the Pre-Launch Report a couple of years ago.
This is where when you upload your APK to the Play Console
test tracks, we will generate a report within an hour of how
your application performs across those different 10 Android
devices that we have in the Test Lab
when a robot is navigating your application for about
10 minutes.
It gives you a report showing you screenshot clustering
and a bunch of insights into accessibility issues
and privacy issues, and also flags crashes.
One thing we recently did is that we enabled you
when you look at a crash cluster in the Vitals section
to be able to see if we have a Pre-Launch Report crash
that we've detected that matches that crash from the field.
This is pretty useful, because with the Pre-Launch Report,
these are test lab devices.
So there aren't the same privacy restrictions in place
as there are with the data from the field, of course.
And so here we are able to show you
the video of the robot interacting
with your application so you can be able to quickly
reproduce that issue yourself.
And we've got a lot more logs and detailed information.
We also did the reverse.
So we now have the ability to be able to see when you're looking
at a Pre-Launch Report, is this a crash
that is already happening in the field?
Or is this one that maybe just was
introduced with my latest APK, but that hasn't actually
reached users yet?
We have over 100k developers using Android Vitals today.
Hands up here.
Who's using Android Vitals?
Cool.
Great.
Thank you guys for using it.
One of the things that I found most interesting
about this project is that we've now
expanded the number of users who are using performance metrics,
or what we call them, like engineering metrics.
So I was previously working on apps on another team at Google.
And I also did that external to Google before.
And one of things I really struggled with
was understanding how these metrics were doing, because you
had to have the latest version of the code
and be able to run the profilers.
And now we've made it so that it's
really easy for all the people in your company
to be able to get access to these performance metrics
and be able to see how you're doing relatively so
to other people and make the case for investing here
versus in other features.
I've shown some case studies from different titles here.
And as you'd expect, you have an Android engineer,
head of mobile engineering, a product manager using it.
But now we have lead mobile product managers
using performance metrics.
We have a CTO, VP of product, and even COOs
using engineering metrics.
And so I encourage you and your teams
all to start using these metrics in your conversations
with your senior leadership and with the wider team.
And if you don't, you're probably
going to get left behind, because quality
across the whole Android ecosystem is improving.
Now, you might think, OK, that's great.
That's impacting those other developers,
but it's not impacting me.
My app's awesome.
Even though it says it has a high crash rate,
I don't believe you.
Well, we looked at the data of users
who experienced a high crash rate versus users
who experienced a low or medium crash rate.
What we find is that users who have a higher crash
rate for the same app leave 52% more one star
reviews or ratings than users with a low or medium one.
And so really, I would encourage you to start using Vitals today
if you're not.
Great.
So what is the future of Android Vitals?
Well, we've built a lot of features,
as you can tell so far.
And so really the future of Android Vitals
is you guys using the product.
And so what we're going to do is right now
we're going to try something different.
And I want everyone to stand up and talk to the people
beside you.
So you're trying to find-- yeah, stand up.
Stand up, and find someone beside you who hopefully you
haven't worked with.
And just introduce yourself.
And we're going to take two minutes for one person
to talk about their vitals and how they're approaching it,
and who in their company uses it, what are the best
practices they've found, and then
switch roles after two minutes.
I'll give you two minutes now, and then we'll switch over.
So I hope you learned something from the people you talked to
and made a new friend hopefully as well.
So I would encourage you to talk to other people
about how they're approaching their vitals.
There's a lot of documentation available online
to help you as well.
And we'll be at the booth outside afterwards
just on the right-hand side when you walk out of this room
if you've any questions as well.
One of the things that you probably
realized talking to that person is
that one of the biggest problems with vitals
today is actionability.
And we've heard you guys and your feedback.
We really appreciate that.
And I wanted to assure you that that is something that we're
working hard to improve.
And we will have more to come on that in the future.
We do believe that the data is valid in terms of telling you
the direction to go, as in like you're
doing good or bad versus other people,
but we think we can do better in terms
of helping you actually be more efficient at fixing
the problem.
But that is a really hard problem.
And we're working on changes to the whole OS itself,
but those things take a while to propagate across the Android
user base, and then to have enough data to be
able to share back with you within the console.
But it's definitely something we are working on.
I want to thank you all very much for joining us today.
And best of luck with improving your vitals.
And I hope I get to share your vitals success
stories in our talk next year.
Thank you.
[APPLAUSE]
[MUSIC PLAYING]