Hello, this talk is about collecting data from people using applied games.
It's called "Trading Accuracy for Enjoyment", because we give evidence for a trade-off:
Applied games, while more enjoyable, reduce the accuracy of data collected, compared against
a practice-as-usual control.
But first, let me talk to you about ice cream.
Across HCI, market research, and the behavioural sciences, we frequently want to collect data
from people.
Often we're interested in questions about them as a person: Personal preferences, attitudes,
beliefs, experiences, or dispositions.
Questions from market research like: "What's your favourite flavour of ice cream?"
or from linguistics, like "What is your grammatical judgement about this sentence?".
I'll call this human-subject data.
We collect this kind of data all the time.
We usually do this with questionnaires and lab experiments.
We tell people to "answer 30 questions about using this software", or "look at this screen
and press the spacebar when you see a black square appear".
The trouble is, methodologies like these are often boring.
Boredom can make it hard to recruit and retain participants.
It can harm data quality, attention, and effort.
Playing games is not boring.
So can we design games to collect data?
This is why people have made games like Sea Hero Quest.
In Sea Hero Quest, you pilot a boat towards targets you remember being shown on a map.
All of your spacial navigation data is collected, and it contributes to a massive dataset of
4.3 million players.
The use of games like these allow more data to be collected from more people, more quickly
and at less cost.
The use of games like these is based on two premises.
First, such games are more enjoyable than the equivalent experiment - so that they motivate
participation.
Second, the data they generate is of similar or better quality to an experiment.
Indeed, data quality work on gamified data collection has generally found that gamification
improves enjoyment, and though results are mixed as to whether it improves data quality,
it doesn't seem to make it worse.
However, gamification, as usually defined, is the use of game design elements in non
game contexts, typically things such as points, achievements and leaderboards.
Fully-fledged games are different.
They are whole games, played as games.
This matters because games as games generate systemic new threats to validity.
They are complex, variant, and in particular, framed and experienced as games.
Two studies have suggested that fully fledged games might harm accuracy, but neither used
a real experimental control.
Therefore, our research question was the following:
"How does the enjoyment and data quality of a human subject data collection game compare
to a practice as usual control?"
I ran a study, and a replication, to answer this question.
Participants started with a demographics questionnaire.
They did a task, then completed an enjoyment questionnaire.
In the task they were assigned to one of two conditions.
Half got to play a digital game.
Half performed a practice-as-usual control task implemented using the same interface
as the game.
Finally, I measured the accuracy of data through the game.
Enjoyment was measured with the Intrinsic Motivation Inventory - Interest/Enjoyment
subscale.
The game I used is called Adjective Game.
It's a puzzle game.
The premise is this: You see a grid of shapes of different sizes and colours.
Your goal is to clear this grid.
Each turn you describe a set of shapes using three words, such as "filled green square",
or "empty blue diamond".
All the matching shapes in the grid are cleared, and remaining shapes fall down, like in the
game Two Dots.
If you clear a group of at least three adjacent shapes at once, you get bonus moves, which
you need to complete the level.
Strategicaly, players need to balance clearing the most shapes now with moves that create
groups of shapes to clear later.
In contrast, the control condition didn't have any of these game features.
It implemented a picture description task, using the same interface.
The task is a sequence of discrete trials.
In each you are asked to describe a highlighted shape.
The visual appearance and interaction is similar to the game condition, but it no longer plays
like a game.
Both conditions collect data about adjective order.
In English, Adjective order is consisitent, but adjective order may differ in different
languages.
Indeed, while English is very well studied, the majority of the worlds languages are little
studied.
By studying adjective order in different languages, we can gain evidence on questions like: "is
there a universal structure to language?"
Ultimately what we are interested in is an individual's judgement, which might be affected
by dialect and upringing.
However, Adjective Order is reliable enough that, for English speakers, we can make a
very good guess at what an individual's quote-unquote "honest" response should be.
Good enough for statistical comparison.
This allows us to operationalise data quality as the accuracy of their adjective orders
to standard English.
In our replication we compared not against standard English, but each participant's own
judgements, elicited afterwards with a questionnaire.
In order to control threats to the validity of this data, which might arise from the game
design, we created Adjective Game using a principled design approach called Intrinsic
Elicitation.
You can read more about how we applied this in the paper.
We expected that in the game we would see higher enjoyment, but lower accuracy.
In both studies we found significantly higher enjoyment in the game.
In both studies we found significantly lower accuracy in the game.
But the game elicited data that was significantly more accurate than random.
In summary, using a game meant that we observed a tradeoff.
We achieved higher enjoyment, but accuracy of the data decreased.
We were concerned that games as games might harm data quality.
We suspected this because of the systemic threats to validity that games might impart.
While previous studies testing gamified experimental tasks found that data quality was maintained
or improved, we found the opposite.
While this suggests that gamification may be better than use of whole games for data
quality, the interesting question is "why?"
Why do games such as this one harm accuracy?
One possibility is that it is the result of increased task demands arising from strategic
play.
Another possibility is framing.
Games played as games may adopt the social norms of gameplay and eschew the social norms
of experimental participation - like careful responding.
Finally, we saw a decrease in accuracy between our two studies.
This might be explained by framing differences between the two studies, but this might also
be history effects from COVID-19, or regression to the mean.
I have gone on to explore the role of demand effects in work that will be published first
in my thesis.
My results are that they can have moderate effect, and are likely partially, but not
fully responsible for the results I have presented here.
Finally, our game was designed to ellicit adjective order - a type of data for which
we cannot give performance-conditional rewards.
Our results may not generalise to cases where performance-conditional rewards are possible.
To test generalisation, further studies on more games and more types of data are needed.
In conclusion, we found evidence for a tradeoff between player experience and data quality.
This suppoorted our intuition that games as games introduce new threats to validity.
It is not yet clear why this is, but framing or demand effects may be involved.
Thank you.