Hi, I am Saeed Safikhani Here I am presenting our work with the title
of : The Influence of in-VR Questionnaire Design
on the User Experience We did this study in the institute of the
interactive system and data science at Graz University of technology
One of the powerful aspects of VR environments that makes them stand out from other virtual
experiences is the high level of immersion Immersion can be described as an objective
characteristic of a virtual environment and could lead to a sense of presence
Although presence is the result of immersion, these two terms are theoretically different.
The sense of presence can be defined as a subjective illusion of being there
This feeling is the key feature of believing in the environment. Breaking the sense of
presence can cause negative emotions in user The moment in which the user loses the illusion
of a virtual environment through awareness of the real world or inconsistency of the
mental model is described as Break In Presence (BIP)
Accordingly, the design of the VR environment should be consistent with the user’s mental
model, and the persistence of the sense of presence should be maintained
A suitable design is needed, from the user interface to the interaction system to design
a coherent experience in VR Several design guidelines follow the traditional
non-immersive design approach when designing a VR environment
2D layout for user interface elements Although this approach has been well developed
and implemented for desktop applications over the years, the question arises whether this
design approach can be in contradiction with the intrinsic properties of VR
This question can be investigated by evaluating the user experience in VR with different design
choices Results of study indicated that 2D interaction
gives a good overview, but speech is faster and 3D is more fun.
The evaluation of the user experience in virtual environments can be done either with subjective
methods (e.g., questionnaires) or with objective methods (e.g., eye-tracking , brain activity
measurements)
Although the use of objective measurement methods is promising, questionnaires are the
most frequently used tool in user experience studies for VR
These questionnaires can be used as pre-, real-time, or post-assessment method
This study used the term out-VR questionnaire to refer to the traditional paper- or web-based
questionnaires that users can interact with outside of a VR environment
the implemented user-interface object which provides interaction with the questionnaire
inside VR is referred to as the in-VR questionnaire
This paper discussed whether the in-VR questionnaire results could also get influenced by the UI
representation
In order to tackle this research question, We developed a toolkit to implement the questionnaire
in the VR applications that can be used as an interface between traditional web-based
questionnaires and the VR environment. Our conclusion from the literature indicated
a positive user response to in-VR questionnaires. we presented the immersive questionnaire toolkit
as a general approach for implementing in-VR questionnaires.
Two factors accompanied the design language of this toolkit to ensure a consistent VR
experience: visualization and interaction The visual representation of the toolkit was
adapted by the overall concept of the virtual scene.
The interactions follow the same rules as the other interactive objects in the environment.
we integrated the possibility of directly importing predefined questionnaires from the
LimeSurvey database into the toolkit. This figure demonstrates the structure of
the toolkit from importing questionnaire data to the post-processing stage.
To evaluate the feasibility and performance of our immersive questionnaire toolkit, we
conducted a user study by integrating the toolkit in a VR environment.
Accordingly, the critical research goals for this study is:
RG1: finding differences in system usability between different designs
RG2: investigating the influence of interaction design on the sense of presence
RG3: finding the difference in task load for different interaction systems
we used the NASA TLX questionnaire to investigate the extent of the task workload
SUS questionnaire for evaluation of usability IPQ to compare the impact of the questionnaire
design on the users' sense of presence In addition to standard questionnaires, we
asked users to fill out open questionnaires regarding their experience to freely reflect
their opinions about each design we recorded the time spent completing the
task (entering the user ID) and playing the game, together with completing the questionnaires
separately In total, we had 16 participants
with ages between 21 and 35
- Most had a university education
- 75% often play video games
- 44% experienced cybersickness before
We provided users, either locally or remotely, a zip file containing a PDF instruction and
the main application
At the start they filled in the demographic and preliminary questionnaires on the LimeSurvey
website
The main application consists of onboarding room and three main rooms
Onboarding-Room familiarize users with the required interactions and objects
Onboarding-Room essential for users without VR experience, or also in the case of remote
evaluations
Consideration of specific Covid-19 protocols for participants attending the university
campus evaluation
Here you can see the gameplay of onboarding room
To reduce the probability of cybersickness through navigation in VR, we used teleportation
as the main locomotion mechanism in addition to physical movement.
The first interaction in the instruction of the Onboarding-Room is teleportation.
Following on from this the user learned how to manipulate the grabbing system and worked
with the handles and the laser shooter to select objects.
At the end they learn about laser pointer. This tool later will be used in 2D layout
On finishing the Onboarding-Room, the user encountered the main game environment, which
we designated as the Questionnaire-Rooms. According to their user ID, the order of the
rooms is determined by the application between all six permutations.
The rooms are identified by the questionnaire layout in there, i.e. 3D, 2D, and web-based.
When accessing this environment, the user should enter a specified user ID. As you can
see in the left figure
After confirming the user ID, the user goes on to play a game as an activity
You can see the structure of the game in the right figure
this is done by throwing balls to hit targets The targets move in the predefined path with
different sizes and speeds The game ends when the user hits all ducks
or reaches the time limit
When the users finished the game, they were taken to the questionnaire stage in the case
of in-VR layouts Here you can see the structure of the 3D layout
questionnaire object in the scene This interactive questionnaire object consists
of three main parts:
1. question text and questionnaire progress monitor
2. the handle of answer selection and range indicator
3. accept and return handle
The user can select the answer with the blue handle and confirm it with the green ones
The 2D layout questionnaire also follows the same structure as the 3D questionnaire
but with similar pointer interaction as in conventional VR applications
The user can select an answer by aiming at the value on the wall using the laser pointer
and pressing the "Accept" button
When each room has been successfully dealt with, a specific environment appears,
prompting the user to remove the headset and return to the desktop to enter the next level
In this study, we investigated the impact of two design alternatives for in-VR questionnaires
and compared them to a traditional web-based questionnaire
Statistical analysis of the results showed no significant difference in the case of SUS.
This may well be an exciting result with different causes, and will need a user study with more
participants for a full evaluation. One of the probable reasons for this, however,
could be that different design approaches did not cause a significant change in system
usability
When considering the respective survey duration, there was a significant difference between
the 3D design layout and the web-based version
The 3D design requires more time for participants to complete the questionnaire, but the NASA
TLX items showed no significant differences
It can be concluded that despite the higher time requirement for in-VR questionnaires,
3D VR questionnaire did not burden users with more tasks.
In contrast, the 2D layout showed a significant difference from web-based on the mental and
frustration items of the NASA TLX
This could be due to several reasons:
(1) There is more task load for users when they fill in the questionnaire in-VR:
This may be a correct interpretation, but in this case, users should face the same condition
with the 3D layout, since the 3D layout takes more time
(2) Participants are more frustrated when using an in-VR questionnaire with the 2D layout
compared to the web-based version: According to this interpretation, users would
not be likely to choose the 2D layout over the web-based version
It was our observation, however, that most users preferred the 2D layout over the others
(3) The implemented questionnaires cannot reflect the correct response of the users
in the VR environment: This should be investigated further either
with sensory data or other questionnaires
we found a significant difference between the 2D design layout and the web-based version
in case of realism
As we explained, the designed environments were the same in all cases.
This means that users should not experience any difference in the Realism case.
This raises the question of the applicability of this type of common questionnaire for VR
studies
This study is the first stage in our journey to design the immersive questionnaire toolkit
in VR The current version of this toolkit provides
the ability to integrate and share data between the LimeSurvey website and the Unity game
engine According to our survey: participants preferred
in-VR questionnaires over out-VR ones However, we need a study with more participants
to reach concrete results
As a future study we will investigate the following hypothesis:
implementing a more consistent questionnaire design in VR could lead to a higher perceived
user experience together with greater accuracy for the survey results