Hey, folks, my name is Konstantin, an engineer on Crashlytics.
I’m here with Sam to talk to you about improvements Crashlytics has made
for games in the recent months.
Games are some of the most sophisticated pieces of mobile software,
and a lot of times they run into issues that are very challenging to debug.
Issues like memory corruption or absent GPU features
or unexpected screen resolutions.
Don’t worry. We’re here to help.
Crashlytics has released a number of features that aim to help your game
reach its full stability potential.
Now, we can’t talk about games without mentioning NDK.
This is the tool of choice for those looking to squeeze the last bit
of performance out of hardware or those that need unparalleled customization
like an artisanal rendering engine.
With NDK, however, it won’t take long to realize
that debugging issues is quite challenging.
The latest Crashlytics NDK SDK has several valuable improvements
that will help you find and solve NDK issues quickly.
These improvements are split into three themes:
Reliability, ensuring that we capture crashes consistently;
Quality, ensuring reports have the necessary information to help you debug your issues;
and Ease-of-use, that is making it very easy to get up and running.
Reporting that a crash has occurred is arguably the most important job
of a crash reporter.
But, as the modern day literary masterpiece that is the POSIX standard, tells us,
“Reliably capturing all crashes in the native world is not so straightforward.”
To enhance Crashlytics’ reliability,
we’ve completely replaced our crash capture internals.
In the previous version of the Crashlytics NDK SDK,
we used Google’s open-source crash capture client called Breakpad.
Although Breakpad was good,
it struggled to accurately capture crashes in certain circumstances
due to its architectural decisions.
So we’ve replaced it with Crashpad.
Crashpad was written from the ground up,
taking into account the many lessons learned from running with Breakpad.
It is the next evolution of native crash reporting with unparalleled reliability.
The immediate benefit of Crashpad is that it’s able to capture not only more errors,
but more categories of errors.
For example, capturing SIGABRT
on Android 10+ devices proved to be quite challenging for Breakpad.
Now, it’s a piece of cake.
To put the importance of this into perspective,
more than 50% of all Android phones are running an Android version 10 or above
and SIGABRT is one of the most common types of crashes.
Crashpad removes this large blind spot.
It's able to achieve this by doing much less within the crashed app.
Crashpad launches a brand new, healthy process
from the signal handler in order to capture the crash.
All of the hard stuff is done outside of the crash context.
In other words, by doing much less, Crashpad is giving you much more.
Reliability, that is.
Receiving a crash report is just the start of the battle.
Decoding what it is trying to tell you is the next big challenge.
Have you ever looked at Android-generated tombstones that contain only
a single application frame or no application frames at all?
This happens frequently for apps that are highly optimized.
The mechanism that generates these tombstones
doesn’t have access to any debug information from your app.
So this is the best that it can do.
Luckily, the Crashlytics NDK tooling
does have access to your application’s debug information.
In the most recent version,
we’ve switched our symbol file format to that of Breakpad.
Now, Breakpad and Crashpad share a bit of their tooling,
including symbol file and crash report formats.
The Breakpad’s symbol file has more debug information that results in higher fidelity
within the application frames of the stack traces,
giving you more clues to where the problem may be.
Compared to our previous format, the Breakpad’s symbol file has more symbols,
symbols that correspond to inline and hidden visibility functions.
The resulting stack traces paint a clearer picture of the path
that the error took through your app.
The Breakpad symbol file also contains Call Frame Information,
information that helps our backend with unwinding the stacks
that ultimately get displayed in the Crashlystics Console.
Without this information, the unwinding mechanism relies on heuristics
that can be inaccurate in a lot of cases.
Ultimately, a more accurate set of application frames within the crashed stack
helps you get to the root of the problem more quickly and easily.
The simple generation mechanism will even work on stripped binaries
in cases where you don’t have access to their unstripped counterparts,
as is the case with a lot of third-party SDKs.
These updates bring huge improvements to reliability and quality,
but perhaps in the past, your custom project structure prevented you
from even trying the Crashlytics NDK SDK.
Do I have great news for you today!
It is now easier than ever to configure the NDK SDK for your project.
Now, only a single property dictates where to look for symbol files.
And the greatest part, though, is that you can specify multiple directories to scan.
No longer do you have to copy libraries that are built outside of your project
into internal build directories.
By the way, if your project has an NDK dynamic feature,
the Crashlytics NDK SDK can now be delivered as part of that feature,
reducing the size of your app.
This is great.
To recap, we’ve increased reliability
by changing the underlying crash capture mechanism in the Crashlytics NDK SDK
from Breakpad to Crashpad.
We’ve increased quality by switching the symbol file format
to the Breakpad’s symbol file.
We’ve made it easier than ever to configure the Crashlytics NDK SDK for your project,
no matter if you have a non-standard setup or even an NDK dynamic feature.
All of these NDK improvements have created a very solid foundation
that we’ve built upon in our Unity offering that Sam will walk us through.
Hi, I’m Sam Edson, another engineer on Crashlytics.
I’m going to talk about how we’ve built upon these NDK improvements
to power crash reporting for Unity.
Unity is a really popular game engine for mobile games,
which Crashlytics has supported for many years.
We’re going to go over three areas where we’ve made improvements to our Unity SDK.
First, we’re going to talk about how we’ve brought great IL2CPP support to Crashlytics
to get more actionable stack traces in the dashboard.
Then we’re going to talk about improvements to issue prioritization
so you can quickly tell where your biggest problems are.
And, lastly, we’re going to talk about crash metadata,
and its tricky problems since stack trace doesn’t give you the whole picture.
Let’s dive right into IL2CPP support, focusing on Android for now.
So a few years back, Unity introduced IL2CPP as an option
for game’s scripting backend.
It is now the default option, which comes with performance benefits.
But for crash reporting, IL2CPP presented many challenges.
Let’s dig into IL2CPP’s compilation process to see why.
As the name suggests, IL2CPP transpiles your C# code into C++ code.
That's a first step of the process.
On Android and iOS, this code is directly compiled on the respective platform
using their build tools as part of the second step.
In fact, when you export your Unity game to Xcode or Android Studio,
you can actually inspect the C++ source files with the IL2CPP compiler generated.
So with IL2CPP as our scripting backend,
let’s talk about what the crash reporting experience was like
before we made improvements to support it.
Looking back, Crashlytics for Unity used to only include the standard
Java-based Crashlytics Android SDK.
Now Unity handles native crashes in an interesting way,
by wrapping them up in a Java exception,
so our standard Java crash reporter can report these native crashes.
This sounds great, but in practice,
the contents of the crash leave a lot of room for improvement.
Looking at the stack trace, figuring out what part of your code caused this
would be really tricky.
The biggest problem is the frames don’t have symbols.
Now you may be able to make use of the memory addresses with some hacking,
but it isn't straightforward.
If you’re like me and you prefer lines of code in your stack trace,
I have great news for you.
The latest version of Crashlytics Unity SDK will come bundled with the NDK SDK
that Konstantin talked about previously.
This brings all those improvements for C++ crash reporting
over to Unity for handling IL2CPP crashes.
In the past, folks have tried to integrate this NDK SDK into their game by hand,
but it was difficult having to modify your exported Android Studio project.
In addition, that workflow didn’t provide a way
to upload your app’s symbol files to Crashlytics,
which gets you those helpful stack traces with the lines of code in them.
To solve this, we’re excited to announce
we’re integrating Android native symbol uploads into the Firebase CLI.
Even when Unity generates multiple symbol files,
uploading them to Crashlytics is just a command line call away.
Also, if you have native plugins in your game,
you can upload their symbols using the same tool.
So, with the inclusion of the NDK SDK and the Firebase CLI,
we get awesome symbolicated C++ stack traces on Android.
Working backwards on our diagram,
we’ve completed one part of our crash reporting story.
We’re able to get crashes for our generated IL2CPP code.
Now here are the raw C++ stack traces from this updated version of our Unity SDK.
This is a huge improvement over the Java-based crashes
as there are line numbers and actual method names.
These improvements will be rolling out to everyone on Android in the coming weeks,
but there’s still one problem.
These frames correspond to the generated IL2CPP C++ code, which you didn’t write.
If you look closely, you can see the frames resemble your Unity C# code,
but mapping it back to C# is tedious.
For example, the first part of each function corresponds to your C# file name.
Before we solve that problem,
let’s take a quick step in iOS since we haven’t talked about that yet.
So similarly, on iOS, we've also made improvements to C++ crash reporting.
Crashlytics Unity on iOS has always had native crash reporting out of the box,
but often crashes would be reported with missing frames in the stack trace
due to Unity framework symbols not being uploaded.
We've revamped the Crashlytics Unity SDK to automatically upload these symbols,
so now you’ll see fully symbolicated frames as shown on the right.
So that covers native crash reporting on both platforms.
This brings us to the process’s next step: Frame Prettification.
We talked previously about how we can see awesome C++ stack traces for IL2CPP apps,
but we’ll still have trouble mapping them back to our C# code.
Well, we’ve introduced new smarts into our pipeline to do that mapping for you.
When you’re crashes go through prettiification,
you’ll get a much clearer picture of where the problem is in the C# code you wrote.
As you can see, you end up with a stack trace that’s much more readable.
This more easily maps back to your code so you can quickly understand/fix the issue.
Stack traces are super helpful for many problems,
and we think this will make diagnosing issues with your app much easier.
Now, let’s change gears and talk about Issue Prioritization.
Once you start getting a lot of crashes,
you’re gonna need a way to know which ones are the most important to tackle.
Customers do this via the Issues List shown here,
which is the front page of the Crashlytics dashboard.
Unfortunately, Unity apps were susceptible to an issue we call overgrouping.
Too much of overgrouping means
unrelated crashes show up as the same issue in this list.
In the worst case, all of your crashes will show up as one mega issue,
that you need to page through to find the real issues.
This makes it hard to see whether issues have multiple root causes, or just one.
Our team’s been hard at work measuring and tuning overgrouping factor for Unity apps
so you can tell when your crash is rare, or affecting many customers.
We now cluster on more appropriate frames,
which gives you a hint for which line in your stack trace that caused the problem.
Now we will target the C# code you wrote, instead of engine library code,
making the stack trace even more relevant.
We really needed the other improvements like C# stack traces
and reliable IL2CPP crash reporting in place to make this happen.
And best of all, these improvements don't just apply to crashes,
they also apply to non-fatal exceptions thrown from C#.
Now I want to move on to one last area of improvement.
Have you ever had a problem that only impacts certain devices of screen sizes?
For example, an issue with some graphics cards
missing a critical feature that your game depends on.
Well, customers used to log this data themselves via Crashlytics custom keys.
Custom keys are great for adding context to a crash.
But now we include hardware and operating system metadata automatically,
so you can easily look for patterns across your sessions to debug those tricky crashes.
This also frees up custom keys for other things.
So that concludes our section on updates to Unity.
To recap, we walked through multiple improvements on stack trace quality,
building on our NDK SDK on Android.
Then we talked about how we could leverage that
to group related issues making prioritization a breeze.
Lastly, we talked about additions to crash metadata,
which makes fixing tricky crashes easier.
As we were saying at the start,
games are a beast and game codebases are complicated,
but we know gaming apps have a lower adoption of crash reporters
than other types of mobile apps.
So, if you want to try out all these awesome features,
navigate to the “Getting Started” pages
and be on the lookout for future technical blog posts.
Thank you.