With all the amazing Android announcements at I/O this year,
great testing tools are critical to Android developers now more than ever.
Hi, my name is Adarsh Fernando,
and I'm a product manager for Android Developer Tools.
And in this video, we're going to focus on what we're doing
to help testing on Android be more reliable, scalable, and productive.
Have you ever had tests passing on your CI server,
only to see them fail locally in Android Studio?
Situations like this can lead to losing confidence
in your automated testing infrastructure,
and more importantly, lead to lost productivity.
One of the causes for this type of inconsistency
is that Android Studio and the Android Gradle plugin
implement different versions of the Android instrumented test runner.
And even small differences in the parameters used to run your tests
can lead to different test results.
To resolve this issue, we’re delegating all tests from Android Studio
to run through the Android Gradle plugin.
We already started this transition in Android Studio Arctic Fox
by moving all unit tests to execute via Gradle.
For example, when you run unit tests
directly from the code editor using the gutter action,
Android Studio automatically creates a Gradle configuration for you
and executes your unit tests via Gradle.
In the latest Canary version of Android Studio,
we’re moving all instrumented tests to run through the Android Gradle test runner.
This helps ensure your instrumented tests return more consistent results,
whether you run them from Android Studio or from the command line.
To get started, make sure you’re using the latest Canary versions
of both Android Studio and the Android Gradle plugin.
In Android Studio, make sure the option to use the Gradle test runner
is enabled in the testing section of the settings or preferences.
When you’re set up, just start running your tests like you normally would.
Android Studio automatically creates
an Android instrumented test configuration for us and starts executing the test.
Under the hood, it’s delegating to the Android Gradle test runner.
And we can see output from Gradle in the test results window.
After the tests have finished, we can then inspect any test failures.
While we've put in a lot of work to seamlessly integrate
the Android Gradle test runner into Android Studio,
we’re also really excited to talk about the integration of a new test executer
called the Unified Test Platform.
The Unified Test Platform, previously called Nitrogen,
is an extensible test executer for running Android tests at scale
from Android Studio and the Android Gradle plugin.
It allows us to make improvements
and add functionality to Android testing in a modular way,
independent of the operating systems and build tool chains that utilize them.
So we can now incrementally deliver features and capabilities
to Android developers more quickly.
Integrating UTP as the main Android test executer was an important investment,
and we're really excited about what this technology
will allow us to do in the future.
But let’s talk about what we’re doing today to take advantage of it.
One such feature enabled by UTP is Gradle-managed virtual devices,
which lets you define devices using Gradle DSL.
During test execution, the virtual devices are fully managed
by the Android Gradle plugin,
from downloading system images and creating devices,
to shutting them down after test's complete.
Gradle-managed devices make it easier to run instrumented tests in a controlled way.
Because of this, Gradle can also guarantee the state of virtual devices.
So test results can be cached and emulator snapshots can be used
to quickly restore devices to a clean state between each test run.
Let’s take a look at how you can get started with Gradle-managed virtual devices
using the latest Canary version of the Android Gradle plugin.
Here’s an example of a device definition in an app module’s build.gradle file.
We’ve given it a name and set the device type to a ManagedVirtualDevice.
We then tell Gradle to use the Pixel 2 device profile and set its apiLevel to 29.
You can use the same device profiles and API levels
that are typically available to you from Android Studio.
Finally, because we want this device to come included with Google’s Play Store and APIs,
we set the source of the image we want to use to Google.
We also say the ABI for the device to x86.
To run tests with the Gradle-managed virtual device,
run the Android test task with the device name,
followed by the build variant you want to test.
Note that because this feature requires UTP,
we also need to pass the property to enable it.
So, to run tests against the debug version of the app
using the Pixel 2 device we just defined, run the following command.
The Android Gradle plugin downloads any system images it needs,
runs the test on the device, and returns the results.
And what’s even better, the plugin utilizes emulator snapshots
to quickly restore the device to a clean state between test runs,
improving the performance and consistency of test results.
You can also specify and run tests across a group of devices in parallel.
Here, we've created an additional device, a Nexus 9, running API level 30.
I can then create a device group.
We’ll name it phoneAndTablet, and include both the Pixel 2 and the Nexus 9.
To run tests across all devices in this group in parallel,
simply use the following command with the name of the group
and the build variant added to the Android test task.
As you saw with Gradle-managed devices,
we now run tests across multiple devices in parallel
to help improve the scalability of your test execution.
And you can take advantage of parallel device testing from Android Studio too,
whether with physical devices, emulators, or a combination of both.
In Android Studio, choose Select Multiple Devices
from the device drop-down.
Check the devices you want to use and click OK.
Now when you run your tests, Android Studio does so across
all your selected devices in parallel.
Results from each device are then clearly displayed in the Test Matrix,
our new test results report panel
designed from the ground up for parallel device testing.
The table view allows a quick glance on which tests are failing on which devices.
And to narrow the results to just the devices you want,
you can filter by device or API level.
You can then select failing tests to inspect its logs
or inspect information about the device.
Test Matrix is also where Android Studio
is able to help you discover rich test artifacts
that can help you more easily debug your app.
One such test artifact we're excited to share
is called Emulator Snapshots for test failures.
Emulator Snapshots is a powerful feature
that captures the state of your virtual device at a specific moment in time
so you can quickly restore that state later.
When executing tests on Android emulators,
we now enable tests to automatically take a snapshot on a test assertion failure.
Let's see how this works.
Here, test are being executed
and the results stream to the Test Matrix tool window.
When a test encounters an assertion failure,
the emulator generates and saves a snapshot and includes it in the test results.
Select a failure to see information about the snapshot that was generated
and restore that state on the test device with a click of a button.
From there, you can immediately start diagnosing the failure by using the debugger.
This opens up a lot of useful possibilities.
For example, if you’re trying to debug test failures that are flaky,
difficult to reproduce, or take a long time to execute,
you can simply use Emulator Snapshots for test failures
to reproduce an issue much more quickly and conveniently.
One thing to keep in mind is that you can only load snapshots on the same device
and host machine that were used to initially capture the snapshot.
To start using Emulator Snapshots for test failures,
add this to your app’s build.Gradle file.
By default, the Android Gradle plugin
generates only two snapshots per device per test run.
But if you need more snapshots in your test run,
just let the Android Gradle plugin know
the maximum number of snapshots to generate using this property.
Alternatively, you can enable and configure the feature
from the command line by passing this property to Gradle.
And best of all, Emulator Snapshots for test failures
works with Gradle-managed virtual devices.
So you can run these tests with the device group we created earlier,
and the Android Gradle plugin will provision the devices,
run tests across them in parallel,
and generate emulator snapshots on assertion failures.
Remember, if you’re running tests outside of Android Studio,
import the results by selecting Run, Import Tests from File from the menu bar
and import the test results protobuf file from the app’s build output directory.
We've covered a lot of new testing features in this video.
To recap, we're making test results more reliable
by migrating Android Studio to run host side and instrumentation tests
through the Android Gradle test runner,
leading to more consistent test results between local and remote test runs.
We’re helping you scale testing by enabling parallel device execution
and streaming multi-device results
back to Android Studio’s new test results panel, the Test Matrix.
We're also integrating a brand new test executer
called the Unified Test Platform, allowing us to more easily develop
scalable testing solutions,
such as Gradle-managed virtual devices, which allows you to define virtual devices
that Gradle provisions and manages during automated tests.
And we’re also leveraging UTP with Emulator Snapshots for test failures,
which automatically generates snapshots on test assertion failures,
helping you more quickly and conveniently reproduce and diagnose tricky bugs.
We can’t wait for you to try all the features we've talked about in this video.
We’re looking forward to sharing even more
about what we’re bringing to testing for Android in the future.