Hi, I'm Julian Frommel. I'm a postdoc at the Interaction Lab at the University of Saskatchewan.
With this video, I present our paper on self-reports in games. We investigated how data
collection in NPC dialogues affects data quality, quantity, player experience, and information
intimacy. I present this work on behalf of my co-authors Cody Phillips and Regan Mandryk.
Self-reports are widely used in games research and development. They are used in basically all
types of playtests and most user studies that measure experience, for example game enjoyment.
They are used in commercial games. For example, Dota 2 or MTG Arena use post-match surveys to
assess if players enjoyed the specific match. The game Until Dawn has virtual therapy sessions.
Players answer questions in interviews with a therapist. The resulting information about
players is used as input to adapt game features. This shows that self-reports
are powerful tools for data collection and that they are actively used in research and practice.
One problem of self-reports is that they are retrospective and therefore do not provide enough
temporal resolution. Displaying them during play is problematic because that interrupts gameplay.
We did address this in earlier work. We proposed
the integration of questionnaires in games with consistent presentation and control.
Then, we expanded this concept by proposing that assessment could be integrated in dialogues with
non-player characters. And this is actually quite close to what commercial games do now. If we
consider the Dota 2 example and just add a speech bubble we basically have an assessment through
NPC dialogues. So we were interested in further evaluating the effects of such an assessment.
We conducted a follow-up study on earlier work to investigate unexplored and inconsistent effects.
In our study, we broadly examined four aspects. We were interested in the amount of data that can be
gathered because it might be that players feel more obligated to respond to a virtual character
than to a faceless UI element. So, NPC dialogues could be useful for increasing data quantity.
In our earlier work, we found that NPC dialogues can be used to validly measure states.
We wanted to confirm these effects and tested whether responses validly measure player states
and traits. This is important to assess if NPC dialogues can be used to gather high-quality data.
In our earlier work, we also found that in-game assessment might be better than overlay
questionnaires in terms of presence but player experience was not better for NPC dialogues in
our other study. So, we wanted to conduct a more comprehensive evaluation of player experience.
Finally, data collection in NPC dialogues is quite unobtrusive, almost like hiding that
we're collecting data. So, we were interested in effects about perceptions on privacy. For that,
we assessed whether players shared more intimate information or perceived information as more
intimate. We implemented a 3D platformer game with a virtual character. Players had to navigate a 3D
level and reach a target flag. During the game, we used in-game self-reports to assess information
about players. For this, we had two conditions: Assessment through NPC dialogues and assessment
through overlay questionnaires. We collected several subjective and behavioral measures.
We compared 166 participants in two between-subjects conditions.
We had overlay questionnaires similar to how you would imagine a questionnaire
and we had NPC dialogue questionnaires that used the Assist Cube as an entity
who actually asks the players the question that we are interested in. What did we find?
Players might be more willing to answer more scale items, suggesting that researchers can gather more
data. In-game responses were predictive of post-experience scores which indicates good
data quality. We found that non-player character dialogues led to significantly
higher experienced meaning. And finally, players perceived shared information as more intimate.
This means that NPC dialogues can be useful for data collection in games. They could be used to
gather more voluntary self-report responses. They result in high-quality data that reflects
the player's experience. And they could make play more meaningful. But players could perceive shared
information as more intimate. So, this has to be applied while considering the player's privacy.
For more information, I refer you to the paper. Please feel free to contact me if
you have questions. Finally, I thank you for your interest in our research