Literature on game design.
A [? crossword ?] of all the major textbooks
that people like to look at.
You'll find sort of three major themes popping up.
And so one is that game design is iterative.
We all like to talk about we have to build things
over and over and over again.
Failing fast.
Finding fun.
There's a heavy emphasis on player experience,
which I don't think would surprise anybody in this room.
And there's also a heavy emphasis
on the major interdisciplinary of game design
in that most game design is done by more than one person.
And in fact, usually many people of diverse
backgrounds and perspectives.
And so a major piece of the process of doing game design
is understanding feedback within this iterative loop.
And when you think about feedback in game design,
many of us probably go immediately
to the concept of play testing.
Because it's kind of like our sort of personal special method
that game design kind of has.
But if you look at broader design practice,
there's sort of these other notions
of other kinds of feedback that happen in the process
and in particular peer feedback.
And so our lab at CMU has been doing
a bunch of work on peer feedback but more
in a pedagogical context.
So how do we understand the process
of doing peer feedback in a classroom?
But then we started to wonder what
does peer feedback actually look like in the industry?
And are we actually teaching it the way that it actually
gets used?
And we found that nobody had really
studied peer feedback in the games industry very much.
And so that's where we set out on this study.
So really we're trying to answer this question of how
do professional game designers make use of peer feedback
within their processes?
And so what we did was a semi-structured interview
study.
We reached out to two game studios.
One was about a midsize.
About 50 employees.
Primarily mobile game studio.
We got 10 interviews from there.
And then the second studio was much smaller.
About 15 employees.
Contract serious game studio where
we got two interviews from.
We did a bunch of iterative development of codes.
Much more detail on that in the paper.
And we talk about lots of things like peer feedback
venues, types of feedback, but what I really
want to focus on in this talk is this peer feedback cycle.
But there's much more about all of this in the paper.
So the peer feedback cycle was sort of this six-stage process
that we arrived at by looking at a bunch of different vignettes
of peer feedback experience.
And it goes through six stages of planning feedback, preparing
to receive feedback, presenting artifacts, exchanging feedback,
processing feedback, and acting on feedback.
And what I want to try to do is dig
into each one of these stages, provide
a little bit of context, and then talk
about how we might use this process.
So the first stage is planning feedback.
And this includes things like first realizing feedback
is even necessary.
Deciding who you want to be reaching out to.
How you want to be reaching out to them.
And also planning your feedback timeline.
So how long is this going to be?
Is it going to be a quick conversation?
Is it going to be a whole week long meeting thing?
And so a representative quote is,
"I try to schedule it so that even if it's
myself and the head of the design team
and like the designers of that game
specifically would participates.
So you're seeing they're thinking
about who is going to be in the room,
and who am I going to reach out to.
And we saw some interesting differences in this
based on who was initiating the process.
So if a feedback receiver was initiating the process,
they often had a goal in mind even
if they couldn't articulate it.
They kind of knew what they were getting into.
But there were some interesting distinctions when other people
started the process.
So feedback providers can start off things,
and they often seem too disruptive.
So a random passer by saying, hey, maybe
you should look at this part of your art.
But there are also people that valued that kind of insight,
because it had no context whatsoever about where the game
was in its current process.
And having that non sequitur feedback
can sometimes help them break creative roadblocks.
And then when third parties, like managers, higher-ups,
would initiate feedback that was often seen as an obligation.
You have to participate, because your boss said so.
But there were some people who appreciated
that those higher-level people might actually
have broader context on where this game needs
to be at the studio level in terms of timeline.
The second stage is preparing to receive feedback,
and we distinguish this one from planning
as essentially this is the concrete actions
to facilitate feedback.
So this is things like creating initial builds that you might
send to somebody, maybe creating surveys
that you might give to your team,
and also reaching out and doing all the logistical planning
and such if that's necessary.
And so a classic representative quote of this
is, "I send the build via link.
I post that in Slack.
Send out an email.
Announce to the company meeting.
And then create a document that anyone could access,
so they could write the feedback on".
It's a particularly well organized person.
Then the next stage is presenting feedback.
And this where they share materials to be viewed.
And this can be anything from just dropping
a piece of art in Slack and saying,
hey, take a look when you get a chance.
Could be actually having a formal presentation with slides
and everything and like a pitch process.
Or it could be reaching out and doing a formal testing
internally.
And an interesting note that we saw
at this stage was the quality of presentation
could often affect the quality of feedback
that would be received.
And designers were very conscious of this.
And so noting that they were very visual in their particular
studio, and if they didn't have slides--
even if it wasn't necessarily relevant to the thing
in question--
they might not get as much engagement
out of their audience.
Then the final exchanging feedback is the fourth stage,
and this is sort of where the magic happens.
It's where feedback goes from providers to receivers,
and then receivers are trying to mainly capture at this stage.
And so a representative quote is, "we'd all play in the space
together and then just give free-form feedback.
Just anything that comes to mind.
And if we're looking for something specific,
we may call it out.
And as somebody that has been running one of these sessions
I'd just have a notebook just writing down
things like crazy".
And there was a lot of comments at this stage
of the social impacts occasionally of peer feedback
where people would potentially start
giving the same kinds of notes or the same kinds of ideas
because everybody else was.
And then also noting that they're
interesting potential power dynamics of--
more junior employees might ignore some feedback
or might not be as aware of what is worth paying attention to.
I'm trying to speed up.
Actually, I should be fine.
So the fifth stage is processing.
So this is where feedback is digested, collated,
might be revised, might be translated from physical notes
to digital.
And then also where things like judgments are being made.
Is that valuable or am I going to ignore that?
Stuff like that.
So representative quote of that.
"The next thing I was doing was processing all the information
and then coalescing it into good action items and takeaways
and ways that we can improve the game that we're working on".
And a really interesting thing that we
noted here is, well, most of the rest of the stages
have variations in formality.
So they have informal methods and formal methods.
The processing stage was almost exclusively informal and very
rarely had an explicit process to find at a studio level
or at a team level.
And we thought that was notably interesting.
And then, finally, the sixth stage is acting on feedbacks.
This is where you execute all the decisions you were made.
You might change something, maybe the whole thing,
maybe a little thing, maybe nothing at all.
And you also might completely scrap your game.
So basically I ended up redesigning the app icon
to match more the visual style but still
have some of the heart of the concept that I came up
with beforehand.
Or occasionally we throw it out and just completely
change everything based on feedback.
And it's hard when it happens, but it does happen sometimes.
And so it's a bit of a whirlwind.
There's a lot more detail on all those different stages
in the paper.
I highly recommend you check it out.
But overall we have this process.
And so I wanted to give a couple quick insights
about how we might use it.
And so we've been talking about potentially using it
to drive intervention design.
So for example, in the planning stage creating digital feedback
portfolios that could help curate feedback given by people
in the past and help you select who's
going to actually be on your team.
For exchanging feedback you could provide guiding questions
to target the feedback that's going
to happen within a particular session towards specific needs.
And then, in the processing stage,
formalizing that process-- that to the moment is currently not
formalized--
and also better understanding how we might structure those.
And then another thing we've been thinking
about is what kind of outcomes do
we want to be looking at if we're looking
to do interventions at these different stages
in different parts of the process?
Are we looking at game success?
Are we looking at employee satisfaction?
Are we looking at player experience
and how the happier game designers create more fun
games?
And then two other pieces of quick future work.
One thing you might have noticed is
we sort of have a small and medium sized studio,
but we have no idea what the indie context looks like.
When you're a team of two people, who is a peer?
Is it just the other person that works for you?
Do you go to a community?
And how does that play into this kind of cycle?
And then the other one that I would
love to solve if anybody here is willing to get
their legal department angry, is the AAA scale.
We've tried.
It's very hard.
But how does scale of a studio really impact
these kinds of processes when your art department is
the size of one of these studios we talk to?
And also it was notable in the two
studios we talked to we didn't hear a lot about crunch.
It just didn't really happen at these kinds of studios,
but we know that is a thing that does happen.
And it would be great to know does
feedback get thrown under the bus when crunch has to happen.
And we just don't have insight on that,
so if anybody would like to help me get into AAA studio
let me know.
But with that I'll thank you and take questions.
[APPLAUSE]
Thank you.
I think we have time for two questions.
So maybe-- I'm Simon Helman.
University of Canterbury in New Zealand.
So maybe I have missed it, but that type
of feedback that you're asking for
is mostly for an individual.
Is that correct?
It's not like a group is presenting,
and that they get feedback.
We defined it as feedback on a work product.
And whether or not it was a team pitching
to the whole studio or one person
pitching to their neighbor.
We didn't really make a distinction.
And I don't-- well, so the vast majority of what we saw was
more that kind of drive by, talking to your neighbor,
kinds of feedback.
So if I had to give it a number of what was more common
I would say that.
But it was usually being received
by at least one person, but sometimes there
were multiple people in a team and one was taking notes
while one was fielding questions so there was
a bit of a difference there.
OK, just a small follow up question.
Did you have any programmers who were searching for feedback?
In software engineering, there's like the formal process of peer
review and something like that.
Yeah, so we didn't have any software engineers by design,
so our goal was essentially anybody
more or less titled designer.
And that's more or less what we got.
At the larger studio we did get a couple of the people
like one was more of a QA sort of position,
and then we did get a little engineer-y in that one.
And then another one was a community manager,
but they also ran play testing.
So there was a bit of multi-hat wearing that we did experience.
But we didn't talk to any specific devs,
but it would be good to think about how the broader studio
also was affected by the sense.
Katherine Girling, [INAUDIBLE].
Thanks for your talk.
I think for all of us who teach game development this
is hugely inspirational.
So I was just wondering, have you thought at all about taking
that methodology, the structures that you find
and adapting it so that it can also be
used in an educational setting.
Yeah, and so Amy is continuing to do that.
She graduated.
She went to Memphis.
So keep following her.
She has this great tool called Peer Presents which actually
facilitates peer feedback during presentations
and kind of works almost like a crit but also a presentation.
It's really cool.
I use it in my class.
I can show you but one thing we are looking at in that trying
to keep that work going, is actually looking more at
that process problem of the digesting feedback
is a thing that often doesn't have a lot of pedagogy for it
already.
And we would love to develop both on the pedagogical side
but also potentially help the industrial side as well.
Thank you.
Thank you very much.
[CHUCKING]
No questions.
Thank you very much.
[APPLAUSE]