XIN LIU: Hi, everyone.
I'm Xin from NetEase games.
Today I'm going to talk about Airtest project,
the next generation automated test team for games.
Two month ago, on JDC we launched here
an open source-based project.
So how many of you here have heard of it?
Oh.
Pretty much.
OK.
Today I'm going to talk about these three topics--
challenges in testing, and how Airtest project
can help you solve these problems; and also
our internal priorities of larger scale real device
testing.
So the first issue is that we have so many games
to test in NetEase games.
These are games we developed and released since 2014.
We have released over 150 mobile games.
And also, we have run several PC games for over 10 years.
So how do we ensure the high quality of all these games?
Another issue is Android device fragmentation.
And because Android system is open source,
and OEMs can customize their own Android phones.
And this graph shows the situation in China.
You can see Samsung, Huawei, Xiaomi.
Each have dozens of Android models,
and each model has different screen resolution,
different system APIs, or even graphic drivers.
in NetEase games, we usually need
to test more than 200 Android phones before releasing a game.
Then let's take a look at the difference of testing
between apps and the games.
The first one is usually our games
are released on multiple platforms,
like Android, iOS; the same version of game
on different platforms.
And sometimes on desktop.
And just now we said Chrome OS, right.
And sometimes Tim HTML5 platforms, console, or even VR.
The next difference is that we got less support
from a third party.
I mean, such as test frameworks, or testing tools,
and platforms.
Unlike apps, Google provide app developers
with test frameworks, like UI Automator, or is [INAUDIBLE]..
And also testing tools are integrated in Android Studio.
Now also we have Firebase Test Lab for app testing.
Another big issue is that games always need more test cases,
because users, there are more content
that can be played in a game than in a app.
So the test cases are increases exponentially in games.
So how do we solve these issues?
We can hire more and more people to do testing.
But hiring more people to do test
on 200 Android phones, that's not tolerable, I think--
even if it's in China.
So automation is here to help.
Actually, we have developed this project for three years
internally.
And last year on Google I/O, I talked with Firebase Test Lab
team.
And we collaborate open sourcing these project.
So let me introduce Airtest IDE first.
It's a desktop IDE.
The right side is the mirror window
of an Android phone connecting with your PC via ADB.
And the middle part is a code editor,
where you can write arbitrary Python code.
And the left part shows the hierarchy
of this, the UI hierarchy of this Unity game,
and also the APIs provided by our test framework.
And when you operate on the screen window,
test code will be automatically generated in the Code Editor.
Let's see the demo.
Yeah.
Click the recording button.
And perform a touch action, it will
generate a sentence in Python.
Also, you can edit it.
Swipe action.
And then we can slip one second.
Also, make assertions of the UI show up.
Then we can run it immediately on this phone
to see if it works.
After you run the tests--
oh, yeah, it worked--
after you run it, you can check the HTML report.
It shows every step of your tests.
If any step went wrong, it will be labeled red,
and you can see the failure point.
You may have noticed that we have two underlying test
framework here.
The first is Airtest framework.
It uses image recognition [INAUDIBLE] knowledge
to locate UI elements.
And then in the use device APIs to perform simulated input.
This is the structure of a Airtest.
We provide users with a simple test API, like simulated input,
like make a insertion.
And then it uses image recognition to locate a UI.
And then we have a underlying abstract layer of platform API.
We unify the simulate the input APIs of different platforms
so that user can run their script
on different platforms, like Android, iOS, and Windows,
and VR.
There is another framework called Poco.
This framework is similar to the task frameworks for apps,
like UI Automator for Android.
But most of the games uses graphic API,
like OpenGL or Vulkan to render the UI widget.
So we cannot directly use UI automator on Android for games.
So we developed Poco.
This is how it works.
We also have a underlying abstract layer
of device abstraction here for different game engines.
We provide each game engine an SDK so
that the test framework can communicate
with the game engine.
We are JSON RPC.
And then we unify different game engines.
So we can write test for different game engines.
Also, the custom engines.
In the morning I saw most of you are using custom engines.
We provide multiple language SDK so you
can implement in your own game engine,
and start using our test framework, and also our tools.
Here is a comparison.
Airtest use image recognition to locate UI,
and use platform API to perform simulated input.
And what's cool here is it does not
require any instrumentation.
Just plug in your phone, and start using.
And we support multiple platforms.
And Poco uses UI hierarchy inspection,
and use JSON RPC to communicate with the SDK in game engine.
Now you need to integrate our SDK.
It usually takes less than 10 minutes.
And now we support Unity and Cocos, those two game engine.
And also Android native app, we implement an SDK
using accessibility service.
So you don't need to integrate anything
into app for Android native apps, and also custom game
engine.
So how do we choose which framework to use?
Here are some suggestions from us.
Actually, in NetEase, we have written
thousands of tests for different type of games internally.
And the advantage of Airtest is obvious.
It requires no instrumentation.
And using open CV to do assertion,
to make assertion of the UI, is reliable,
because it checks exactly what users are expected to see.
And for Poco, since it uses UI hierarchy inspection,
it can be more accurate, especially for cases of 3D
object when the object change their orientation in games.
And this Poco is similar to test frameworks
for apps like UI Automation.
And it covers the what is missing for games.
And we released these two project GDC two month ago.
And we got 6,000 downloads, and over 1,000 stars
in this two month.
And please try our project, and star us on GitHub.
But it's started.
We have two more new features in this two month.
First we support iOS.
Actually, we have supported iOS last year,
but the performance is not good enough to be released.
Let's see this demo.
Connecting your iOS with your x code.
Then you can operate the screen just like for Android phones.
And this is one of our games.
Then you can run the tests on this iPhone.
That's it.
The back panel of this iOS support
is originated from Facebook web travel agent.
But the web travel agent, the problem is that it's too slow.
Simple touch action can take as long as one second,
and it's not tolerable for games automation.
So we optimize most of its API last month.
And now the frame rate can be up to 15 FPS in the Airtest IDE.
Another feature is we support web pages automation.
You can open the Chrome browser in the IDE.
And then you can operate on the Chrome.
And then code will be automatically generated.
Yeah.
The [INAUDIBLE] is based on Selenium framework.
Also you can use Airtest API to make
insertions to run the test, and then generate the report.
Since Selenium is quite immature in web pages automation,
we have no reason to reinvent it.
So we just wrote a plugin for Airtest IDE
to help recording this web pages automation.
I think it will be useful if you are releasing game
on HTML platform.
Until now we talked about Airtest project.
You can use Airtest IDE to record and run the test
on your PC and on your phone.
But how do we test hundreds of Android phones?
The first thing is Firebase Test Lab.
Firebase Test Lab is a test service provided
by Google Firebase team.
And what do we do collaborating with their team
is we support running Airtest and the Poco script
on Firebase Test Lab.
And you can use Airtest IDE to bundle your test
script into an APK.
And then you can upload it to the FTL web page.
And then you can start a instrumentation test,
and use the cloud of devices provided by FTL.
And this is our internal device farm.
We have set up a device farm of 200 devices--
200 Android devices.
And we have written now the number could be 2,000 scripts.
And we run those scripts of different games every week,
because our game are updated weekly.
And let's see how it works.
It's just [INAUDIBLE] and around the test
generated by Airtest IDE.
Yeah, that's it.
As for our roadmap for this automation project,
we intended to support more clients--
iOS, Android Emulator, and web pages.
These are in better version.
You can try it now.
And feel free to fire us a issue if you find any bug.
And also we are Hybrid apps, and UnrealEngine is in the future.
Also, we supported Android TV.
But I'm not sure about Chrome OS.
But maybe it works.
And also another important thing is
we want to build up the open source
community for this project, and I hope more and more developers
like it, and get involved in this project.
Yeah, thank you.
And any question?
AUDIENCE: You mentioned you're using
the accessibility service.
Did you have any issue with that when
that was restricted recently?
XIN LIU: When it what?
AUDIENCE: The accessibility service.
It was restricted for apps that were using those launchers,
or for any apps that were trying to access the reverse
[INAUDIBLE] tapped in large.
Those are things that were being restricted because they
were being taken advantage of.
Have you guys seen any impact from that?
XIN LIU: For UI testing of native apps,
we don't see any restriction.
I think it can do what UI automation or [INAUDIBLE] do.