♪ [MUSIC] ♪
[MIKE] Alright, everybody, it looks like we're three past the hour,
so let's go ahead and get started.
Thank you all for coming to our Unite Now session today.
Test Smarter, Not Harder: Game Simulation and Anipang 4.
My name is Mike Carley, and I'm a Senior Product Marketing Manager
here at Unity on the Game Simulation product.
And I'm very lucky to be joined by my coworker, Christina,
who can do a fantastic intro.
Hi, yes, I'm Christina. I'm a Senior Product Manager
on the Game Simulation team here at Unity.
Perfect. So without further ado,
let's go ahead and jump right into our session.
We have a great agenda here today. We're going to start
by going over the different challenges
surrounding testing games today.
We're then going to introduce you to Unity Game Simulation,
what it is, what it does, why we think
it's a pretty fantastic solution.
We're then going to go over our case study of the day,
going over how Game Simulation helped the Korean developer
SundayToz with their hit game
And last but certainly not least, we'll give you a future-looking view
of where we plan to take Game Simulation,
both today and into the future.
Let's jump right into it. We'll start today with a challenge
that I'm sure is familiar to all of you, and that's the fact
that launching a quality game is hard.
And by definition, quality here needs to be
a very all-encompassing term.
It needs to mean your game works correctly and is bug-free.
It needs to mean your game looks great,
It needs to mean your game is engaging
and offers players a challenge that they believe they can overcome.
Testing is a great solution to help with this before launch,
but ultimately, testing is costly, it's time-consuming,
and it's inefficient.
And if you've ever had to test a game pre-launch,
and you know all the different bevy of tests that you need to run
to ensure your game's quality, you quickly realize
why this level of very ingrained and elongated testing
can really do damage to your ability to compress
your development timelines.
There are tests like game unit testing,
which are very binary tests that we run on bits of game code.
Things like if a bullet hits a player,
does that player get injured?
There are QA playthrough tests which typically amount
to monolithic spreadsheets that a human playtester
needs to sit down and go line-by-line through
executing the requested interactions,
a very time-consuming and tedious process.
There are balance validation and optimization tests.
These are tests that ensure your game is challenging
and engaging. If you have a shooter game,
you don't have a combination of gun variables
that destroy all other guns or enemies in the game.
If it's a racing game, you want to make sure that your car variables
don't add up to a single car or collection of variables
blowing all other players out of the water.
So ensuring your game is balanced, challenging, and engaging
is a very important set of tests that we need to focus on.
And with next-generation consoles coming around the bend,
performance testing your game to make sure it works
and behaves correctly across all different devices
under all different loads and stresses of players
is an extremely important set of tests.
Testing is not a small portion of your game development timeline.
And there are some great solutions on the market today,
but they all have their limitations.
There's not a one-size-fits-all solution out there
to model their game, whether it's the game economy
or the different game parameters,
to model their game and hopefully, make some predictions
about how those different combinations of variables
will actually play out in the actual game.
And because you're relying on a human to hopefully encompass
all the different combinations of variables in your game,
these models can often be oversimplified.
And because these models are oversimplified,
we can call into question the predictive value they have
and what these combinations of variables will actually
net out to in the actual gameplay experience.
Human playtesting is a great option, especially when it comes to
the more qualitative aspects of your game:
Is this fun? Am I being challenged? Will I come back and play again?
However, human playtesting is extremely costly and not scalable.
If you want ten simultaneous playthroughs,
you need to pay ten individual people
to sit down in front of a device and play through a level ten times.
There are additionally some automated solutions
on the market today, but they're very limited in what they do
and they're very niche. It's usually one very small
testing use case that they address
and they don't really have plans to expand the types of use cases
that they do address.
So there are some solutions on the market today
to help automate and streamline some of the testing
you have to go through, but as you can see,
they all have their drawbacks.
That's why we built Unity Game Simulation.
Unity Game Simulation provides an affordable way
to automate and scale game balance and QA testing by empowering you
to simulate millions of game playthroughs and interactions
in the cloud.
So that's a great, really tight marketing statement,
but let's look at what Game Simulation looks like in practice.
How can Unity Game Simulation actually help you
as a game developer or designer as you're trying to launch
the most quality game possible?
Unity Game Simulation is designed to be a very easy,
intuitive product to pick up and use.
The first step is to go to our website and download
the Game Simulation package and upload the build of your game.
This will allow Game Simulation to place counters
throughout your game which are used to track the output metrics
throughout each simulated playthrough.
These counters can be increased, decreased, or reset at any point
of the simulation, and the great part is
you don't have to wait for the simulation to complete
to actually view the results on these counters.
You can check the results of these counters mid-simulation,
either at a single moment in time or over a predefined interval,
which is really powerful.
You can then use our intuitive web dashboard
to easily parameterize these simulations
and, of course, kick them off.
What you're doing when you're kicking off a simulation
with Game Simulation, one of the most important
capabilities you're leveraging is what we call grid search.
Grid search is a capability built directly within Game Simulation
that will automatically generate and test
every conceivable combination of game parameters you have.
When we compare it to something like
a human-built spreadsheet model where we're hoping the human
considered every possible variable combination,
with grid search, we're removing that hope.
We're basically saying, "We will absolutely locate,
generate, test and validate every conceivable
parameter combination within the game."
So you know we're leaving no stone left unturned
when we're doing these simulated playthroughs.
You can then run these simulations at cloud scale.
You can scale testing not only, of course, by running them
in the cloud at a much faster rate,
but you can also run parallel simulations.
You can run 2, 3, 5, 10 simulations at once at the same time,
which will not only generate results quicker
than these traditional testing methods,
but because you're running them at cloud scale
and you're running them over hundreds or thousands
of millions of playthroughs, you're going to eliminate
those outlier results and really get a really trustworthy set
of quantifiable results from these simulations
that are a lot more accurate than what could be generated
by an oversimplified Excel spreadsheet
or a human playtester.
And these reliable quantitative results can be viewed
and chopped up and executed in multiple different ways.
You can view the results for an individual simulation
or a playthrough, you can aggregate results
over thousands of playthroughs,
and you can even integrate game simulation with your existing
analytic system to further extend the power
of the data-driven decision-making that you're trying to make
as you tweak your game, test the tweaks, et cetera.
And for those who really want to supercharge the amount
of operational agility that Game Sim uses
and lends to your organization,
you can also leverage our public APIs
to seamlessly slot game simulation into your existing build pipelines,
CI pipelines, or with any other custom functionality you have
built into your game development operational workflows.
You can also use these public APIs
to actually kick off simulations as well.
If you're really developer-focused
and you want to automate the kicking off of simulations
or run it in a more headless manner,
instead of using the web dashboard, you can certainly leverage our APIs
to do that.
When we go back and look at the current suite
of testing options on the market today,
you can hopefully, now see that Game Simulation does offer
some pretty unique benefits when compared to
these three existing solutions.
As I mentioned before, when compared to spreadsheet-based models,
Game Simulation will deliver much more accurate results,
not only because we can guarantee that we're accounting for
every variable combination possible,
but we're also running these at cloud scale
over thousands of playthroughs so you're really sure
that you're not making decisions based on outlier data.
Another more nuanced advantage is that with Game Simulation
you're running a simulation through your actual game code,
your actual game. You're not trying to represent
a proxy of your game with figures and equations
on an Excel spreadsheet.
So being able to actually simulate and test on your actual game
is a massive advantage when it comes to comparing Game Simulation
to spreadsheet-based modeling.
When compared to human playtesting,
Game Simulation is far more affordable and scalable.
We currently charge $1 per simulation hour
for Game Simulation, which is a lot more cost effective
when compared to the hourly rates of human playtesters.
We can also run multiple parallel simulations at once
and run thousands or millions of game playthroughs,
which is a scale that absolutely can't be achieved
with human playtesting.
Last but not least, when compared to the automated solutions
on the market, we have some great market validation
that Game Simulation can help with,
QA use cases and game balancing use cases.
And we're only going to be expanding the amount of use cases
and ways to use Game Simulation from here.
When compared to the other automated solutions on the market,
they really can't compete as far as the reach and breadth
of the things Game Simulation can do.
Because Game Simulation is Unity-native
and we have our public APIs, we can really supercharge
the amount of agility that you can do by allowing you
to tweak your game, test those changes, enact those changes,
test the next round of changes, et cetera,
and make that feedback loop extremely quick and tight
so you can really make agile changes to your game
based on the results of your simulations,
which is something you certainly can't do with a bolted-on
third-party integration.
As you can hopefully see, Game Simulation does have some
really great distinct advantages over the current options
on the market today, to automate or compress testing timelines.
And before we dive into our case study of the day,
I wanted to call attention to two other great case studies
that we've written up and placed on our Game Simulation website:
Furyion's
as well as iLLOGIKA's
Both of these customers used Game Simulation to balance,
and optimize, and deliver the most quality game possible on launch.
But the main takeaway from this slide
is that Furyion's
iLLOGIKA's
and as you're about to see,
Game Simulation is not custom-tailored
for any specific game genre.
Game Simulation can help no matter the genre of game you're building.
So we encourage you to try Game Simulation,
even if you don't see the exact type of game
that you're looking to build reflected in this talk track today.
That's a high-level overview of Game Simulation,
what it is and what it does. And with that,
I'm very happy to pass it over to Christina
to dive into our case study of the day.
[MIKE] Christina? [CHRISTINA] Awesome. Thanks, Mike.
SundayToz is a leading mobile games developer in South Korea,
and they've created over 20 mobile games to date.
One of their most popular games is called
which is a popular puzzle game that has been downloaded over 35 times
since it's 2010 launch.
Anipang 4
And for this game, as is common for a lot of puzzle games,
the SundayToz team needed to design countless new levels
to satisfy player demand.
On average, the team releases 20 to 30 new levels every two weeks,
and making sure that the quality of this amount of content is good,
puts a strain on team resources.
They needed to find innovative ways to engage players, quickly playtest
and check the results, and then make revisions from each new level
before release. SundayToz needed to exponentially increase
the number of tests they could run
so they turned to Unity Game Simulation.
Here, SundayToz really wanted to answer two questions
using Game SImulation.
The first, they called difficulty analysis,
and the question they want to answer is, "Are the levels
the right difficulty? Are they too hard or too easy?"
The second, they called error detection and the question here is,
"Do the levels have errors? Are they playable?"
To answer the first question of whether the levels
are the right difficulty, SundayToz developed
an autoplay mechanism.
Using this playthrough mechanism, they run tests on each level
with a set of input parameters.
These include the given number of turns,
the number of certain block types, mission count, and many more.
The output metric is the failure rate of each level.
After the simulation completes, SundayToz determines
whether the output metric is within a desirable range.
And if not, they adjust the parameters,
for example, by reducing the number of turns
and rerun the simulation.
Once the failure rate is right, the level is deemed
to be the right difficulty. The next question is
whether levels have errors.
SundayToz flags simulations where the autoplay stops playing
before using all of the given turns,
and if this happens, SundayToz identifiies the cause of the error,
which may include wrong mission count,
wrong object placement, or crashes.
Then after fixing the cause of the error,
SundayToz reruns simulations to again verify the change.
Game Simulation has helped SundayToz increase the precision
of their difficulty predictions and reduce the time it takes
to detect errors.
SundayToz can now run 600 playtests every two to three weeks
when new levels are released.
This represents a five-to-ten times more test per level
than manual efforts did before, while saving hundreds of hours
of developer and designer time.
All this testing is paying off.
Anipang 4
And with 850,000 monthly active users,
SundayToz faces more and more demand for new content,
and therefore, more and more testing needs.
Donggeon Kim, who's the technical director at SundayToz,
really values the flexibility that Game Sim gives to his team
on how to allocate their time.
Game Simulation can take the tedious parts out of testing
so developers and designers can focus on content creation
and the fun factor.
By increasing exponentially the number of tests that they can run,
Game Simulation helps ensure a great gameplay experience.
So where does Game Simulation go from here?
There's four main areas that we're tackling.
One is bot creation. We want to help you manage
and create automated playthrough mechanisms
using existing tools.
The second is metric optimization.
Given all the data that Game Simulation can generate,
we want to help you find the perfect combination of parameters
for your desired metric output.
In addition to game balancing, Game Simulation can also extend
to QA needs.
We want to ensure that your UI looks and behaves as intended,
and for multiplayer games, we want to help you stress and load test.
For all these tests, we want to help you run them seamlessly
on a daily, weekly, or continuous basis.
With that, I'll hand it back over to Mike.
[MIKE] Perfect. So as we wrap up this session and we'll, of course,
take any questions that might have popped up during the session,
the one takeaway we want you to leave with
is that you can test better with simulated playthroughs
by increasing the accuracy, efficiency,
and scale of your testing efforts, especially across the use cases
we support today in game balance and QA.
If you're interested in any of the future-facing features
that Christina just went over, please email us
at the Game Simulation email alias you see here.
In fact, if you have any questions or concerns you would like
a follow-up or a deeper dive on Game Simulation,
please reach out to us. We're more than happy to answer
any questions that you might have.
More relevantly, if you want to jump in
and give Game Simulation a try, please sign up at our website
that you can see on this website: unity.com/products/game-simulation
Your first 500 hours of simulation are completely free of charge
to allow you to get in, use the tool, play around,
and see what difference Game Simulation can make
with your game development and with your testing timelines.
That was our efficient and hopefully impactful
Game Simulation session.
We'll pause now for any questions that might have arose
during this session or any questions you'd like to ask now.
Excellent. Feel free to use the Q&A pod to answer any questions
that might have arose, and Christina and I will be happy
to address those, otherwise, thank you so much for your time
and joining our session this morning. We really appreciate
your willingness to learn some more about Game Simulation.
Alright. Christina, this is probably a great question for you:
"Can we train the ML-Agents by using game simulations?"
[CHRISTINA] We have a separate team, ML-Agents,
which is actually a very close partner team.
I'm happy to direct any queries into how to use ML-Agents
or how to use it concurrently with Game Simulation to that team.
Essentially, what'll happen is you can create a bot
using whatever method you want, including ML-Agents,
and all you need to do is bring your own bot to the simulation
to have it run the simulation for you.
[MIKE] Perfect. Second follow-up question from Dan:
"Would this work in a non-game environment?"
And the answer is yes, we actually have an entirely different team,
an entirely different product for non-gaming applications.
That product is called Unity Simulation,
and we have applications in different non-gaming verticals.
Dan, if you are interested in that
I'm going to put my email in the chat pod here.
It's going to be mike.carley@unity3d.com.
If you would like to learn more about the industrial applications
or the non-gaming applications of the simulation technology,
please shoot me an email and I'd be more than happy
to connect you with the industrial side of things.
[CHRISTINA] The other question from Dan is, "It seems like the games
have specific goals, so what if
you have a game that is more open-world?"
As long as you have a way to quantify
the metrics and the parameters,
it doesn't really matter what the type of game is
or if it's open-world versus
deterministic or anything like that.
What's most important is just that you're able to have integer
or float values.
[MIKE] Excellent. Again, you have that Game Simulation,
that Unity 3D alias that I encourage you all to reach out to
with any follow-up. You can also reach out directly to me
via my email that I put in the chat.
More than happy to continue discussions and answer questions
in a more in-depth way.
But again, we really appreciate everybody who could attend today,
and reach out with any other questions or concerns.
And hopefully, when we meet up this time next year,
Game Simulation has done some really cool
and transformational things to your testing timelines.
Wonderful. Thanks again, everybody, for your time,
and have a great day.
♪ [MUSIC] ♪