Hello everyone! My name is Daniel MacCormick
and on behalf of my colleague, Dr. Loutfouz Zaman,
I am going to be presenting our work called Echo:
Analyzing Gameplay Sessions by Reconstructing Them From Recorded Data.
To begin with, let's talk a little bit about evaluating games.
By nature, games are a very iterative process.
We bring participants in, they play the games, and we use their feedback
to make the game better.
We then repeat that cycle.
When we bring participants in, it's called a playtest.
During a playtest, we often record their screens to generate video footage
that we can analyze later to make sure we didn't miss anything.
This can be very useful but at the same time, it can be very cumbersome
since you can only watch one video at a time.
So, recently there has been a lot of research into the area of game analytics.
Game analytics are great because they're very fast.
They are able to show a lot of data really quickly
but in doing so, you often abstract away the game to a different form.
And so the game can take the form of just a series of data points now.
In doing so, you lose a lot of the game context
that helps inform why players made the decisions that they did.
In doing so, you're sort of trading off speed for context.
We can sort of see that here.
On the left side, we have a great tool called Playtracer.
This abstracts the game into graphs of nodes
which is great and shows a lot of data very quickly.
But you of course lose what the game originally looked like.
On the other end of the spectrum, something very similar to video footage actually,
is Fortnite's - the commercial game - replay system.
Using this tool, you are able to analyze Fortnite matches very well
because you have all the original effects, graphics, and so on.
But in doing so, you lose a lot of the speed.
Then we have something in the middle, like PlayerViz here
where you get some of the game context like these primitives
representing a bit of the level,
as well as the analytics overlaid on top.
So, what we wanted to do with this work is we wanted to try to find a better way to
bring both the speed and the context together,
and sort of marry the best of the game analytics and the video footage.
In doing so, we developed a tool called "Echo".
And so with Echo, we are using the game's original graphics.
So, this is a car racing game.
We're reconstructing it from the recorded data.
And so while doing so, we're able to interact with the data,
view it from any angle, speed it up, and slow it down, as we see fit.
Echo has two primary components.
It has got the recording system and the visualization system,
with the log files as sort of an in-between.
We had three main focuses when developing the tool.
We wanted to focus on: interactivity, efficiency, and game context.
So again, we've been looking to try to provide a balance of speed and context,
but also allow the data to be interacted with.
So, you can customize the viewing to make it easier to understand what is going on.
We can see that in action here.
Right now, we're viewing the player's original camera but through Echo.
So, we're able to speed it up.
And we're also able to show a bunch of other planes at the same time as well.
Previously, we just had the one plane but now we have multiple players together.
We are also able to change the camera and view it from a different perspective
and then speed it up.
And so, we have all of these features of interactivity,
while also representing multiple sessions at the same time,
using the game's original visuals in order to provide the most context we can.
After developing the tool, we of course wanted to perform an evaluation.
In phase 1, we brought 10 participants and they played both the plane and car games
which you have now seen.
And generated Echo log files and video recordings.
These became the input to phase 2,
where we then brought 16 participants in to analyze both games.
They analyzed one using Echo and one using video footage,
counter-balanced of course.
While they were analyzing, we tasked them with recording interesting observations
during the analysis of the game that they were looking at.
These could be either positive or negative.
This was designed to emulate how in games user research, we often develop
a list of key positives and key negatives
that can be used by a game's developers to improve the game going forward.
Outside of this requirement, there was no specific workflow they had to follow,
as we left it very open-ended.
We noticed 3 distinct workflows actually come out of that,
with the 3rd being the most interesting and sort of the general design for Echo.
Which is that the participants started by watching all of the data together -
so all of the cars and all of the planes - then looked for extremes or outliers.
So like a car that went off course or a plane that crashed into the ground.
And then after they identified an outlier, they would hide the remaining datasets
and then focus in on the outlier to try to identify what happened.
This is really how we see Echo being used, like a top-down approach.
We also got lots of great feedback in terms of what features they liked and disliked,
and then many suggestions on improving the tool going forward.
We performed a NASA-TLX questionnaire comparison
and the score was significantly higher for video footage than it was for Echo.
Along individual dimensions, frustration was the only one that was significant.
We also asked the participants to rate the tools on ease of use,
efficiency, likelihood of future use, and overall.
Echo was significantly higher in all categories except for ease of use.
The primary limitation is that we compared against video footage.
This was useful for the open-ended nature of the study
but it would also be good to perform a study in comparison against a tool like
Playtracer or PlayerViz.
Additionally, we focused in on a single type of game to keep them relatively even -
- in this case, vehicle racing games.
But it would be useful to see how Echo performs with first-person-shooters,
or puzzle games, or fighting games - all sorts of different games.
And then of course, we can continue to improve Echo
based on the suggestions we received.
So, in conclusion, we developed a tool called Echo in an effort
to try to bridge the gap between game analytics and video footage analysis.
In doing so, we wanted to try to bring the speed that analytics provide while also
keeping the game context in the way that video footage does.
We received a lot of very promising feedback from participants which shows
that Echo has a lot of potential but there is still lots of room to expand and improve
on the tool in the future.
Thank you very much for listening!