Hello and welcome to my presentation titled Examining Design Choices of Questionnaires
in VR User Studies My name is Dmitry
We as researchers often use VR for user studies and frequently our measurements rely on subjective
self-reports; for example questionnaires. Often users leave VR to fill out questionnaires
on a pc or paper and have to return into VR for the next task.
The switch between realities creates a BIP that can cause participant’s distraction,
and negative emotions which in return can lead to biased results.
Therefore, it seems natural that the administration of questionnaires in VR should improve the
quality of a user study.
When designing our own studies and considering applying inVRQs we were surprised that this
assessment method received low attention in the literature.
This indicates that the administration of self-reports in VR user studies requires some
considerations in order to retain validity. To build an understanding of the current practices
and to lay the groundwork for future development, we based this work on the following research questions:
RQ1 How are questionnaires and individual question-items applied in contemporary VR user research?
RQ2 What are the considerations around – and
possible advantages of – administering questionnaires in VR?
RQ3 What are appropriate design choices for presentation and interaction with in-VR questionnaires?
These research questions are addressed in a series of 4 studies which I present in the
following slides.
With regard to RQ1, we conducted a literature review to investigate common procedures of
VR user studies We analyzed both the system design and the
study design of 123 publications between 2016 and 2019 from CHI, CHIPLAY, VRST, IEEE VR
and IEEE 3DUI. 110 of 123 publications use questionnaires
15 use inVRQs. And 7 described them in more detail
The realizations of inVRQs differ substantially in their presentation and interaction methods.
The majority of cases used presentations that contextualized the questionnaires in the VE,
either attached to the users, or anchored in world space.
Their participants used predominantly native controllers to interact in VR, directly followed
by free-hand interaction. These results indicate that there is no shared
agreement on administration of questionnaires in VR.
To augment the insights gained from the literature review, we conducted an expert online survey
that evaluates general proceedings of VR user studies and attitudes towards inVRQs.
We developed a custom survey to capture the experiences with inVRQs as well as potential
problems to consider. For recruiting expert participants, we contacted
of authors from the literature review and advertised the survey on social media.
74 experts filled out the survey. After filtering, our sample consisted of 67 responses.
The responses to the survey indicate that experts appear to have a positive attitude
towards inVRQs. Our expert survey did not highlight a clear
standard method for presentation. Both our literature review and expert survey
demonstrate that there is no common standard for using questionnaires in VR user studies.
Thus, regarding RQ1, the state of the art for delivering questionnaires in VR user studies
is quite heterogeneous. There are experts who have a strong opinion
against embedding questionnaires and argue that inVRQs are not usable and could bias
the self-reports. In response to RQ2, the broad range of reasons
provided for or against using inVRQs indicate technical challenges, implementation effort,
fear of biases and participant overload.
The results from the literature review and from the expert surveys are heterogeneous
and do not allow for a definite answer of the validity and realization of inVRQs.
With regard to RQ3, the goal of this design study was to identify the most usable interface
design to further investigate the validity of inVRQs as a measurement device.
We implemented four variants of inVRQs. The interfaces differ in two dimensions: anchoring
as world-referenced or body-referenced and interaction modality as pointer or trackpad.
The world-referenced UI is anchored in the VE and users can freely move around the interface.
A body-referenced UI is attached to the hand-held controller.
Our interface supports five common types of questions.
The study has a 2 × 2 within-subjects design where users filled out inVRQs using the four
different versions of the interface. Each condition consisted of a questionnaire
that comprised five common knowledge questions with each question types represented once.
After each condition in VR, the participants took off the HMD and filled out a paper-based
System Usability Scale. 10 male game developers with prior experience
with VR participated in the study. The SUS scores and the completion times clearly
show the world-referenced anchoring with the laser pointer interaction is easiest to use
and therefore the best candidate to investigate the concerns raised by the experts.
To contrast the usability of inVRQs against outVRQs, we conducted a user study with 38 participants
The participants shot balloons in a VR archery game and then filled out inVRQs as well as
questionnaires presented on a notebook.
As the design study revealed, we implemented the inVRQs with world-based anchoring and
laser pointer interaction. The outVRQs were realized using LimeSurvey.
It was presented on a 15” notebook with external mouse
As subjective measures we assessed presence, task workload and usability.
Also we logged the task duration and interviewed the participants at the end of the experiment
For presence, the results show positive ratings
of general presence, spatial presence and involvement.
There was no significant effect between inVRQs and outVRQs on any IPQ subscale.
Also the variances of the responses did not differ between the conditions.
The duration of filling out the questionnaires
was around 6.5 minutes with inVRQs slightly but not significantly slower.
Nevertheless, in the interviews, five participants perceived filling out the inVRQs as faster
and two had the impression that questionnaires themselves are shorter.
On the UMUX, the participants rated the usability
of both questionnaire types positively. The inVRQs received a lower rating of 77 in
contrast to 86 for outVRQs.
The participants experienced low to medium
workload on the NASA-TLX with higher physical demand and effort of inVRQs
From the qualitative interviews, we found
that most participants enjoyed the VR experience and that the inVRQs were easy to use.
A few highlighted that with inVRQs the study feels more fluent and avoids BIPs
Rougly a third stated that inVRQs were entertaining. And if the participants would have to repeat
the study the majority preferred to do the question-asking in VR.
The high ratings on the IPQ and the qualitative statements indicate that the game provided
a high sense of presence.
With the UMUX, we measured high usability scores for both questionnaire types.
According to Bangor the scores are in a highly acceptable range, discarding the usability
concerns of the experts. However, the outVRQs were rated higher with
a medium effect size. A possible explanation was given in the interviews:
Sometimes, the participants had to repeatedly click on a UI element for selection.
On the TLX, we measured a comparatively low workload. However, the physical demand and
effort are significantly higher in the inVRQs condition. This could be attributed to the
fact that the participants were standing and using the VR controller in mid-air rather
than sitting at the desk with a resting hand. One participant even sat down on the ground
to fill out the inVRQ.
The comparable duration is consistent with the TLX ratings for temporal demand and allows
to infer that answering questions in VR does not affect the duration of a user study.
Subjective self-reports are frequently used in VR user studies and administered in the
physical domain. This can lead to a break in presence, disrupt the immersive experience
and bias the responses. To overcome these biases we investigated how
to embed questionnaires in VR from 3 perspectives: literature, experts and users.
From the literature analysis, we identified 15 instances of inVRQs. These few examples
differ substantially in visualization and interaction, emphasizing the lack of validated
surveying procedures in VR user research. For a comprehensive understanding of the advantages
and challenges of inVRQs, we conducted an online survey with 67 VR experts.
To explore presentation and interaction modalities suitable for inVRQs, we conducted 2 user studies,
in which we first identified world-anchoring and pointing as most adequate design choices
to administer questionnaires in VR. We then contrasted inVRQs against common screen-based
outVRQs. Although the results show lower usability
and higher physical demand and effort of inVRQs, the ratings are within a tolerable range and
the majority of participants stated a positive attitude towards inVRQs.
Based on the results from the 4 presented studies we conclude:
(i) researchers should apply inVRQs in their user studies,
(ii) pointing and world-anchoring are usable ways to realise inVRQs,
(iii) participants prefer using inVRQs over OUT VRQS and
(iv) researcher should be aware of slightly raised physical and mental demands when using
inVRQs.
In a follow up study we showed than inVRQs can drastically reduce the break in presence
Similar to the establishment of standardized questionnaires that are empirically validated,
we propose moving towards standardizing questionnaire implementation and presentation methods in VR.
Thank you for your attention. For details, we refer to our paper
If you are interested in trying out VRestionnaires, have further remarks or questions please contact
us using the displayed email address.