Samantha: Thank you for joining us.
Obviously we would have really liked to present this in person at CHI, but times have changed,
so thank you for coming to this virtual discussion session.
Myself, Pejman, Guenter, and Atiya are just going to be talking a bit about our paper
exploring the relationship between playtesting reports and the impressions of critics on
videogames.
So, a little bit of background for this work.
Our real motivation here was that there's a lack of research published about the impact
of games user research on the quality of published games.
And this is for a couple of different reasons.
One is that GUR data for commercial projects can be very difficult to access because of
not only NDA concerns, but a lack of organization, particularly for small studios and similar
things like that.
It's also really hard to describe what the quality or success of a videogame is.
There's lots of metrics for that.
Sales figures, obviously, critical reception, reception from users, and so on.
So our goal was to study how playtesting could help identify issues that will impact the
perception of a game once it is released by critics and users alike.
Particularly for indie studios that have less structured user research divisions, where
it can be very difficult to deal with this relatively small amount of user research data
that they have.
There have been some similar works investigating things like the impact of player reviews on
a game's reception, the return on investment of UX evaluation, and so on.
So, our study was consisting of basically looking at 3 case studies across 3 different
genres.
One's a collectible card game, one's a platformer, one's a top-down shooter.
And, the timeline of this was that the production of the games including playtesting occurred
around 2015 to 2016.
The games were released, then, a few years later, we collected reviews for these games.
We analyzed them, we coded them based on the features that reviewers commented on, and
then we did a comparative analysis to see if we could find some relationship between
the features that reviewers commented on and the quality of those features in the game.
And whether those features were identified as potentially problematic by those playtesting
reports that were done during development.
So in a nutshell, what we found, and what we were really looking for, was gaps between
those playtesting reports and critical reviews.
So essentially what features those playtesting reports missed out on, if you will, that were
identified by critics as issues.
And what we really wanted to see here was the role of playtesting methods and study
design in affecting the coverage of a playtest.
You can see where we're going with this.
We kind of noticed similar patterns in the games that we looked at.
Where, the shooter game for instance, the tests that were conducted were focused on
first-time user experience.
And a lot of the things that critics mentioned that were missed by the playtest were related
to long-term play.
So things like the variety of content, or levels, or game modes, or story available.
All things that obviously wouldn't be the focus of a first-time user experience test.
So what can we learn from this?
Again, 3 small case studies, all indie games, all without any kind of centralized structure
for a user research division that would allow for more rigorous playtesting and playtesting
designs.
What can we learn from this about playtesting design in general?
Well, some of our key takeaways were that features related to a game's core mechanics
were discussed really frequently in game reviews.
So playtesting should be advised to focus at least partially on the quality of core
gameplay mechanics, or what's making a game unique.
It's not just about player learning, or first-time user experience, it's about those core mechanics.
And then, aspects outside of those core mechanics can also be really important, depending on
the game.
So if a game's intended for long-term play, content variety, which is something that you're
not going to see from a playtest that only spans a few hours, as we saw with the shooter
game, can be really important.
You need to consider how playtesting protocols will specifically address individual features.
And we have some guidelines present in the full paper that relate individual playtesting
methods to the features they can be best suited to address.
We also, in each of these cases, saw that playtesting didn't focus on creative direction.
Things like, aesthetics, sound, and so on.
And the importance of these features in reviews is really hard to ignore.
When we were coding these reviews, as we saw with Game A for example, things like sound
can be really important to critical reception when you're looking at the package of a whole
game.
And the last point that we're going to highlight here is that playtesting reports tend to focus
on problematic aspects of a game.
But, when you look at the critical reception of a game, you obviously also see reporting
on positives and what a game's core appeal is.
So, if we can report on these positives in playtesting reports as well, it can help developers
to identify the features in their game that are the "sellers" if you will.
The things that critics and players will see as a draw to that game.
And it may help them to focus on further development of those features, polishing those features,
and validate some of the decisions that are made.
So the crux of what we're trying to say here and what's elaborated more in those guidelines
in the paper, is that whatever a developer or the research team is focusing on as key
areas of critical importance post-release, need to be the focus of playtesting pre-release.
It can't just be about general testing focused on first-time user experience or what have
you.
There needs to be a more critical analysis of what a game's weak points are, what potential
strong points need validation, and then playtesting methods need to be tailored to focus on those
strong points.
So that's the paper in a nutshell.
I think we're going to throw to the panel discussion now.
Audience member: Okay, I have a question.
You mentioned a focus on creative direction.
Most often, during our playtesting, we're maybe doing concept testing, or we're doing
usability tests, and so that kind of creative direction isn't really typically a focus.
You said you had some guidelines on that.
Could you speak to that just a little bit?
Because it seems slightly out of scope.
Samantha: Right, so, in the paper, when we talk about creative direction.
Things like graphics, art style, animation, sound, and so on.
The suggested methods that we talk about are a bit related to concept testing.
So, looking at A/B testing earlier in development for things like different sound designs, art
styles, and so on.
Looking at having some opportunities in focus groups, maybe after say a playtesting session
for a multiplayer game.
Making sure that some time is allocated to discuss things that relate to a game's aesthetics
and creative direction.
Obviously, for matters like this, you ideally like to have a much larger group of people
to answer questions about art style because such things are so subjective that you want
to have a larger group.
For indie studios obviously this can be really difficult to do, so having a larger scale
online survey if it's feasible, can be a good option.
Otherwise, asking those questions, even in a small group, just to get an indication of
what people's first impressions are, is certainly better than omitting them entirely.
Pejman: One thing I would add here, regardless of the method, I think one part that was quite
interesting for me, was that it's also difficult to discuss those creative directions with
the design team.
Especially if you are the external party, just evaluating their game.
And also, your playtest is not really equipped to answer some of those questions.
One interesting discussion that sometimes comes up with colleagues either on the user
research side or the design side is, also, how much user researchers need to get into
discussing the design direction or creative direction.
Some colleagues, they think, or they believe, that anything related to design and creativity
should be left to the game designers or creative directors.
But also, some other colleagues think that user researchers need to go beyond just reporting
the experience.
And also maybe provide some suggestions.
In terms of modifying those creative directions.
So, it's kind of interesting, if any of our other participants here can comment on their
view on this.
Audience member: From my experience, and the way we work where I'm at, is, user research
actually does have a direct role in providing suggestions, in all of our reports, for whatever
issues that we're uncovering.
So I think it would depend on the type of tests.
If it was a concept test, then that would be expected, to dig into it a little bit more.
But on a usability scale, with smaller numbers, I'd caution the team.
Like, yes, we have six people here, and this is the feedback we got.
But in terms of creative direction, specifically, with six players, I'm not going to make suggestions,
based on that.
Pejman: Yeah, I totally agree.
And one thing that's also interesting to consider is the relationship between user researchers
and designers.
Because, if it's a company that the user researchers are embedded in the design team, I think those
discussions on creative direction can happen more easily.
Guenter: One point to add here is that perhaps there is a relationship between the design
and usability and experience of the game.
Because the design can also affect, for example, the feedback which is perceived by players.
So, that might be also something to consider here.
So the design might affect how a player is perceiving the feedback.
Could hide something a little bit more, or it can highlight some feedback to some extent.
Samantha: Do we have any other questions from the audience right now?
I think otherwise we might move on to discuss a little bit in general about the guidelines
presented in the paper.
I think, before leading into the discussion of general guidelines, for me at least, one
of the most interesting takeaways from this in the guidelines and what we saw in our analysis,
was the dichotomy between things that are noticed with a short-term playtest versus
a much longer-term engagement with the game.
That sort of emergence of things that simply will not come up in that shorter-scale usability
playtest.
At least, was something that I found quite interesting, simply because running longer-term
playtests and diary studies and the like, with those you run into the complications
of not being able to control the testing environment, necessarily.
Especially if you're a small studio that doesn't have lab space available to have people consistently
coming back.
Looking at, if you have a developer that's running on a shoestring budget where that
sort of thing isn't feasible, maybe there's something that can be done in terms of expert
evaluation.
Especially to look at assessing the long-term appeal of a game.
Pejman: The part that I probably want to put some emphasis on is that, when working with
developers, sometimes they have this misconception that, "Oh, yeah, I did one playtest on our
game, so it should be fine", or "We did two tests", or "We showed it to our friends and
family, so we should be fine".
And there are still many other questions that could have been answered, using different
methods that we didn't do, or that the designers, or developers, didn't want to do.
So, I think that's the main takeaway for me, that one session or two sessions with a specific
focus is obviously not going to reveal all the issues with the game.
Atiya: The type of test that's run for a game really does have to be specified for that
game and what its mechanics are like.
With Game B, the test that was run didn't show a lot of the issues that a lot of reviewers
had, because it's a game that would have benefitted from a longer test.
It's really interesting how, even though these were all indie projects, and all the tests
that were run were small-scale, in many of the cases, that was still enough to highlight
the main things that reviewers would have had an issue with.
Pejman: I think one interesting thought could also be how much user researchers need to
advise on different types of testing.
Like, again, looking back at what we did in 2015 or 2016, the focus of these tests were
motivated by the developers.
So they basically told us that, "Hey, I'm wondering about the tutorial of our game.
Can you test that?"
Where, now in retrospect, I should have gone back and said, "Yeah, we tested the tutorial.
But do you also want us to run some tests to check on things such as level variety,
or game modes?"
So, how much do we listen to what the developer wants from us, to evaluate, and how much do
user researchers need to direct developers in different types of testing that are available.
Audience member: I'm on an embedded team, and so, regarding the options, I think there's
more, like you were saying before, implications for developer teams becoming more educated,
perhaps.
On options that we can offer, like, when you're mentioning longer-term diary studies, or longer
play sessions, or things like that.
Pejman: That's a very interesting point, in terms of educating developers.
I think that educational piece is the key.
And maybe, or hopefully, this paper may have some contribution in that space.
Just to highlight things that could have been identified easily but didn't.
Audience member: Yeah, and I feel like that's part of what's happening in games user research
as a whole, right?
We're still trying to get our methods out there to more people.
Pejman: Thanks everyone for joining.
Atiya: Thank you!
Pejman: I hope this at least sparks some interesting discussion with the team!
Audience member: Yes, thank you for your presentation.
Appreciate it.
Samantha: Thanks for joining us everybody!