Hello, and thank you for watching our presentation. I’m Samantha Stahlke, and together with
my colleagues Atiya Nova and Pejman Mirza-Babaei from the UXR lab at Ontario Tech University,
we will be presenting our work on an automated playtesting tool.
So, we all know that playtesting is a critical part of games user research. To answer questions
about player understanding, challenge, enjoyment, and pretty much every other aspect of user
experience, playtesting is essential. But, there’s a problem – playtesting is
expensive. You’ve got overhead for your testing team and lab space, and of course
you have to pay those pesky participants. Then you’ve got to analyze all of the data
from those participants. And the next time you want to test, you really want to find
new participants that will be seeing the game with fresh eyes. Plus, more playtesting usually
happens later in development - when substantial changes may be difficult and expensive to
make. In other words, testing with humans is inconvenient.
So our work is motivated by this question: Can we take some of the burden off of playtesting
with humans? Can we use AI for playtesting? Using AI to play games is certainly not a
new concept – board games like Chess and Go, and more complex digital games like DotA,
have been an AI testbed for years. AI has also been used to test and improve
games. King, EA, and Ubisoft have all explored in-house solutions for automatic evaluation
of things like level difficulty and combat systems. Most of the tools created in this
space are proprietary or heavily game-specific, though Unity has also released a toolkit for
general reinforcement learning that can be used to develop in-game character AI or testing
tools. We looked at all of this and we wanted to
make a tool of our own for automated playtesting. We wanted our tool to be generalizable for
different games, accessible to designers without a technical background, and freely available
for anyone that wanted to use it. And so, we created PathOS.
PathOS allows designers to simulate player navigation in 3D game environments with AI
agents. We want our tool to help designers answer questions like whether players will
get lost and quickly compare player pathing between alternate versions of level geometry.
PathOS isn’t meant to replace human testing, but rather provide some approximate data without
the time and expense of a full playtest. For instance, PathOS might help in between playtest
iterations while refining level design. We made PathOS as a free extension for Unity.
If you’re interested in checking it out, you can download PathOS on Github.
Our agents implement a simple expert system based on models of how players see, remember,
and make decisions with game information. Our model of player sight uses simple culling
tests based on level geometry and apparent size to determine what an agent should see.
Our memory model means that agents can forget or mix up information about where objects
are in a level, to help mimic the mistakes a human player might make. Lastly, agents
have customizable personality profiles that determine their preferences in what to interact
with. For instance, an agent meant to imitate a cautious player will be more likely to avoid
objects that are flagged as hostile. PathOS agents navigate a level by taking the
information available in their memory and creating a list of possible destinations.
These destinations are scored based on the agent’s simulated personality, before a
high-scoring destination is picked to use in pathfinding. This process is repeated as
the agent moves in the world, creating a rough prediction of player movement.
To support developer workflows, we created a user interface so that designers could “see”
what was happening inside an agent’s head in real time. We also created utilities for
simulating many agents at once and reviewing results via path visualizations and heatmaps.
We conducted a user study for PathOS with 9 game developers, who we asked to use the
tool while designing two levels for a simple RPG-type game. Our participants were very
positive about the utility of the framework, finding its predictions useful in refining
their designs. Several participants commented on the potential time and monetary savings
of the tool over traditional playtesting. We also identified several areas for improvement
in the tool’s user interface and proposed additions to its functionality.
In our future work, we’re hoping to evaluate the accuracy of our tool’s predictions.
We’re also hoping to test the tool with a commercial developer, implement the improvements
we identified, and look into working with machine learning.
Thanks again for watching our presentation. You can read more about our implementation
and user study in the full paper. We look forward to answering your questions!