- Good morning.
I'm Jie from CWI Amsterdam.
On behalf of my co-authors, I'm going to talk
about the study developing and validating a questionnaire
for measuring social virtual reality experiences.
Recently, virtual reality has become alternative
to audio communication, like telephone,
or 2D video conferencing like Skype.
But VR experience is generally described
as being isolated in front of real world,
or from the other human beings.
So to reduce this kind of sense of isolation,
social VR is a type of system
that invites multiple users
to enter the same virtual immersive environment
and to interact
with the virtual representations of each other,
such as Facebook Spaces,
which embodies users' movements onto a avatar
and users are invited to use nonverbal cues,
such as hand gestures,
and even facial expressions alongside verbal communication.
So in this way, it hopes to reduce a sense of isolation
and it resembles even go beyond face-to-face interaction.
While social VR systems are promising
to resemble face-to-face interaction,
it remains a challenge how to measure and understand
such a experience.
There are many existing questionnaires
for measuring, for example, quality of interaction,
social interaction, or immersive presence experiences in VR.
But for a single experiment,
we cannot use all of these questionnaires.
It's not feasible.
So the challenge for us is to combine
all this kind of knowledges into one single questionnaire
that we can use it to measure social VR experiences.
So this challenge leads towards two goals of our study.
The first one is to develop a social VR questionnaire,
and the second one is to validate this questionnaire.
So we validate this questionnaire by conducting
a comparative studies across three conditions;
so social VR condition, Skype condition, and face-to-face.
The first contribution of our work
is the development of a social VR questionnaire,
and the questionnaire combines both
well validated items from existing questionnaires
and items developed by us through user-centric studies.
So I'm going to explain more about the items later.
And the second contribution
is the validation of this questionnaire.
We use photo sharing as a use case
because it's a activity we do almost every day
to maintain our important relationships.
And photo sharing can happen at different times,
so such as Instagram,
so people access into a fixed online archive
to share photos, and can also happen at the same time
between people co-located at the same place,
such as face-to-face sharing,
and even they are separated by distance,
so like Skype sharing.
So among all these types,
we choose to focus on co-present sharing
because it's dynamic and it's improvisational
kind of interaction, and we would like to understand
how we can measure or understand this kind of interaction
in virtual reality.
So some related work.
So, in the past 20 years,
self-assessment questionnaires
has been widely developed and validated to measure presence,
immersion, these kind of experiences in VR.
However, these questionnaires focus very much
on single user experiences in VR, and they focus very much
on presence and immersion dimensions.
And in our study we (coughs) sorry.
In our study, we aim to understand the nuances
of social experiences in VR
by combining knowledges from two fields.
So one is we call technology-mediated social communication
and another field is the existing field
of presence and immersion in VR.
Here, I give a overview of the methodology we use.
So we adopt user-centric approach
to design the questionnaire.
And we conducted a comparative study
to validate the questionnaire.
So in the following slides,
I'm going to briefly explain the process.
So to develop this questionnaire,
we started with a series of user studies
with photo sharing users, virtual reality experts,
and the UX researchers.
First, we conducted a context mapping,
which is a research method invited users
to be experts of their own experience,
so we provide workbooks and generative sessions
to facilitate users to reflect and observe
their use of experiences.
So in our context mapping session,
the users mapped 12 typical experiences of photo sharing.
Based on the results of context mapping,
the expert participatory session mapped
20 categories of relevant social interactions
that can happen in both real world and in virtual reality.
And next, we designed a online questionnaire.
For clustering the 12 experiences
based on the 20 interactions,
and we invited 20 UX researchers to participate.
This is one example of the online questions.
So the UX researchers were asked to select
among the 20 interactions which can elicit
the specific experiences in the question.
We conducted correspondence analysis
of the clustering questionnaire,
which resulted in three categories,
or three components of photo sharing experiences,
namely quality of interaction,
presence immersion, and social meaning.
And we cross checked these three components
with the related work, so with existing literature,
which resulted in 33 items of questionnaire.
In the next few slides,
I'm going through briefly the questionnaire items
and to show you where these items are from.
The first questions to ask the users to self report
his or her own emotions and the intensity.
And this questions belongs to quality of interaction,
and we adopt a easy to use pictorial emotion reporting tool
developed by Vastenburg.
The rest of the questions are using five point Likert scale.
The question items are either derived
from well validated questionnaire,
or composed based on the typical experiences mapped
in the context mapping session.
So question two to 12 is about quality of interaction.
And question 13 to 23 is about social meaning,
and the rest is about presence and immersion.
And next, we use this questionnaire
in our comparative study.
So we invite people to do photo sharing
in three conditions,
as an initial trial to validate this questionnaire.
We recruited a total of 26 participant pairs,
so they are either friends or colleagues.
So the experiment is within-subject design
in three experimental conditions counterbalanced.
In the face-to-face condition,
two participants were sitting together
and showing each other photos on their smartphones.
And in the Skype condition,
participants were sitting in two different rooms,
so they share different picture
on their smartphones via Skype.
And in the Facebook Spaces condition,
we uploaded the photos to the Facebook Spaces
and participants, they were sitting in different rooms,
but they enter the same virtual space to share their photos,
so the photo is represented in a way
similar to smartphone display.
After each condition,
every participant filled in one version of our questionnaire
and after all three conditions,
we conducted an audio recorded semi-structured interview
to get more idea about their experiences.
We used Exploratory Factory Analysis to better understand
the important factors of our questionnaire
and to ensure the items are meaningful and not redundant,
we only took the items with higher factor loadings.
so we can see the criteria here.
And this step helped us to reduce the items from 33 to 24.
And we further tested each set of items
for internal reliability.
And our final items set show
a very high reliability coefficients.
Here, we show it's just kind of example
what kind of results we can get from the questionnaire.
And this analysis is based on the final 24 items.
So for quality of interaction,
face-to-face photo sharing, of course, is perceived
to be much better than the two other conditions.
And in terms of presence immersion,
the social VR system is perceived to be more immersive
and resulted in high sense of presence
than Skype conditions.
And for social meaning,
social VR systems is comparable to face-to-face conditions,
which is most of them are higher than Skype conditions.
For the interviews, so we have a quite extensive discussion
about the interview results in the paper,
but here I give you some highlights.
So in general, participants feel
separated in screen-based communication such as Skype,
and it's difficult for them actually to associate
the avatar in Facebook Spaces with real human being,
so they pay more attentions
to the voice of the virtual representation, the avatar
rather than the gestures of facial expressions of avatar.
But they do feel physically and emotionally closer
using social VR system.
To conclude, our questionnaire can provide researchers
with a tool to measure reality-grounded activities
in social VR, and the avatar-based social VR system
has a potential for providing experiences
comparable to face-to-face activities.
In the future,
we would like to explore more social VR use cases,
and we also want to continue collecting data
to validate the questionnaire.
And we may also want to adapt some objective metrics,
such as physiological sensors
to automatic capture the social VR behavior.
And the social VR questionnaire
can be used in other use cases.
So, I give some examples here.
And currently, we are trying out the use cases
for health care,
so we want to try patient-doctor communication,
especially when the patient has mobility problems.
And we are also looking into the movie watching experiences
in social VR at the moment.
And I give you an example of how to use our questionnaire.
So we can just simply replace activities
in a questionnaire item
by the intended activities of yourselves.
So, our questionnaire is available on GiftHub.
This is end of my talk, I'm happy to take questions.
(audience applauding)
- [Dylan] Hi. (clears throat)
Sorry.
My name's Dylan, I'm from Stanford University.
I am wondering if your questionnaire
captures or if you looked at the nature of the relationship
between the people and how that might influence
the overall experience.
For instance, close friends versus colleagues
that you see at work, it might be a different experience.
- Yes, we try to because we ask
in the background information about the participants,
if some of them have friendship about five years,
some of them are like a couple.
But we don't have enough sample each sub-category,
but we do see some trends,
like when they are close friends, or when they are couple,
they do have some like when they judge the interactions
or closeness, they might give higher scores on that aspect.
- [Woman] Hi.
I'm (mumbles) from KU Leuven.
I was wondering, are you familiar with the social presence
in gaming questionnaire?
- Yes, I think some question items are adopted
from Jellets, the author is called Jellets.
- No it's not Jellets, it's Yvonne de Kort,
and Poole's and Eiselstein.
- No, I didn't look into that.
- Okay, you may want to look into that
because they also have theoretical constructs
on relying social presence,
which might be interesting for you.
- Okay, thank you.
Can I ask you to write down the name for me?
(woman laughs)
Thank you. - I will so.
And I have a second followup question
that it seems very interesting,
but to validate your questionnaire,
you may need to have a wider sample
and also check for discriminant validity
and not just convergence validity,
which you show with your high Cronbach's alpha.
- Yes, that's our plan for the future,
so we want to collect more data using this questionnaire,
so we are now working on doctor-patient scenario
and also movie watching scenario.
And we are also going to use this questionnaire
like news reporting scenario,
so people gather in a news room to watch news,
real news together,
so we are planning for this experiment at the moment.
- Okay. - So I hope this can
help us to externally validate this questionnaire.
- [] Yes, very well.
Thank you.
- I have a question.
So, did you try looking into understanding
the differences or similarities
between sharing experiences
in asynchronous versus in synchronous manner?
- I didn't look into this aspect,
but that's a discussion we had in the paper,
so what about sharing online and sharing face-to-face?
But through our interviews with the participants
before the experiment, we do find,
so face-to-face photo sharing is more enjoyable
because it involves more interaction than online sharing.
- More questions?
No?
Okay, so let's thank our speaker.
(audience applauding)