It is a great pleasure to have the opportunity to present our work at VRST2021.
Our work is entitled “Virtual Reality platform for functional magnetic resonance imaging
in ecologically valid conditions”.
It was performed jointly between the Laboratory of cognitive neuroscience of EPFL and the
Human neuroscience platform of the FCBG.
First, let’s define fMRI.
MRI is the leading non-invasive neuroimaging technique in both clinical and research laboratory
settings.
MRI is performed using a large magnet and allows to get images of brain anatomy and activity
of a person laying inside.
While very powerful, MRI come along many constraints: First, MRI technology involved a 3 Tesla magnetic
field, which is 10 000 higher than earth magnetic field.
This field disrupts the functioning of most electronic devices and attract metallic objects
toward the magnet.
Second, the acquisition of fMRI comes with a strong auditory noise of around 110 decibels.
And finally, the acquisition of high-quality images requires the participant to be static
in the magnet.
fMRI is central for human neurosciences.
fMRI allows neuroscientist to access to the brain activity corresponding to a specific
percept, thought or action.
For instance, the activity of the visual system allows to decode the kind of visual images
a participant is seeing: a face, a house, an object or even a word!
The same applies for the sounds we hear or the body part where we are touched for instance.
Of course, this is not limited to the 5 senses but also to complex activities like imagination,
emotion or even consciousness!
Unfortunately, to achieve a good Signal-To-Noise ratio, neuroscientists need to use very standardized
stimuli similar to desktop-based experiments!
So to summarize, the normal MR environment is
stressful due to magnetic constraints and auditory noise.
Is also artificial due to restricted movement and experimental constraints.
So we propose to create a virtual environment that will reconstitute the MRI room and, within
this environment, to recreate the natural sensorimotor contingencies using embodiment,
presence and naturalistic interaction to create ecological condition for fMRI.
In this presentation, we will present the VR-MRI platform, a combination of technologies
allowing together to create such a virtual environment.
So our platform is built around three main components.
While presenting these components, we will briefly give a word on the hardware we choose,
but it is important to note that our platform is built around the principle of scalability
and is not limited to this specific equipment.
So the first component is the visual display, a MR-compatible stereoscopic googles with
large field of view to maximize immersion.
We went with the VisualSystem HD from Nordic NeuroLabs which is what is the best at the
moment concerning MR-compatible VR googles, in term of resolution and field of view, plus
it integrates an eye tracker and has lots of adjustable parameters as Inter Pupil Distance
adjustment and Diopter correction.
The presence of an eye-tracking technology in the stereoscopic googles is a must have
as most of the fMRI experiments requires to ensure that participants are awake with eyes
open.
Second, a motion capture system to provide a simple way to animate objects and interact
with virtual objects as a user.
In our implementation we use six Qualisys Oqus 500+m cameras.
And third, some hand-held response device to track user’s hands with the motion capture
system, fixing its posture and preventing fingers movements while providing a visual
representation matching between real and virtual world.
Our component is home-made and composed of tennis balls that the participant is asked
to hold during the experiment.
On these balls we fixed a support for the motion capture reference.
There are also three physical buttons per ball that are useful for more classical interactions
as when a precise and accurate response time is needed during a task.
So now that we have our set-up, how do we build ecologically valid fMRI experiments?
For this part we will follow the diagram of the flow of events during an experiment, with
on the left what occurs in the physical environment, and on the right, what happens in the virtual
one.
So first of all, it is important to have a virtual model of the physical room to merge
the physical and virtual worlds.
Then, an important point for embodiment is to match the participant’s actual and virtual
appearance.
Concerning clothes, participants are asked to wear dark pants, a white t-shirt, and black
shoes, to match the virtual avatar clothes.
Concerning appearance, our framework allows multiple modifications on the virtual avatar
to match the participant, as gender, skin color, and waist size.
Once this is set-up, the participant is placed in a supine position on the MRI bed and equipped
with the stereoscopic googles, the response devices, and additional rigid bodies placed
on his knees.
The simulation then starts, and the participants see the replicated MRI room through the stereoscopic
googles, matching what he would see from his position while lying on the MRI bed.
This gave the impression of looking at the actual MRI room, but through optics.
Then, during the entry of the participant into the scanner, the movement of the virtual
viewpoint follows the movement of the MRI table as it entered the magnet.
Importantly, the viewpoint progressively rotates from looking up at the ceiling toward looking
down at the feet.
This gives a realistic first-person perspective view of someone lying and looking at something
in front of him while still being in a suitable position for MRI acquisition.
This is followed by a familiarization phase where participants are given time to observe
how the virtual body moves according to their actual movements regarding their knees and
their hands.
During this familiarization phase, the magnet slowly fades to invisibility to give an open
environment impression while keeping the sense of being present and located in the scene,
as well as increasing the available visual field for stimuli presentation and interaction.
These steps are then followed by the experimental protocol.
You can see an example here of a cognitive task and some interactions.
A key point here is the implementation of a checkpoint system allowing the experimenter
to bring the participant back to a previous point of the experimental flow in case of
unexpected problems.
Indeed, for fMRI acquisition, any unexpected problem causes to stop the acquisition and
restart the entire acquisition block, thus a checkpoint system is essential.
When the experiment is finished, the exit from the virtual environment consists in the
reverse of the scanner entrance.
The participant viewpoint moves progressively toward the exit position and the supine viewpoint
while he is physically moved towards the initial position onto the MRI table.
Here I am going to briefly present the results of the validation pilot.
To evaluate the overall subjective experience in the scanner, participants answered to a
post-scan interview. We asked for the experience of presence, the
agency of actions performed, and if the head tilt was noticed during the MRI entry and
exit.
We found that most of the participants felt present in the virtual environment, felt agency
of their actions, and did not notice the head tilt, consistently with the hypothesis of
a strong immersion in the virtual environment.
Within the MR scanner, we asked 4 questions to evaluate the effect of a first-person virtual
avatar.
Participants were either embodied within a sex- and size-matched virtual avatar - or BODY
condition - or within a color- and size-matched box - or OBJECT condition -
The BODY condition was preceded by an embodiment induction using interaction with virtual buttons.
We found that participant significantly felt more embodied in the avatar, felt more in
control of the virtual body, and noticed less the head tilt in BODY condition, showing that
having a body to identify with increases immersion into the virtual environment.
We also replicate neuroimaging and behavioral results.
Evaluating the effect of head movement on image quality, we found that measures of movement
were on average less than one voxel.
The data removal due to peak movement was less than 10%, thus below standard data rejection
in fMRI.
The brain activity corresponding to perceptual, motor, cognitive and embodiment manipulation
replicated previous research in basic neuroscience literature.
Finally, we found that mental rotation of a displayed character provided faster reaction
times for back-facing orientation, in agreement with literature.
We also found that a 3D character gave rise to lower reaction times, suggesting that a
realistic character made the task easier.
Overall, we validate the efficiency of our platform, having reproduced and extended basic
neuroscience results in ecologically valid condition.
In sum, we show new opportunities for research. Any screen-based experiment can be reproduced
with the VR-fMRI platform.
Sensorimotor contingencies can be studied without significant movement artifact.
And experiments can be made more ecological.
We also provide a few recommendations to use the platform.
Adapting the instruction and establishing a progressive immersion using entry procedure.
Separating the experimental blocks by smooth transitions and using participant embodiment
to stabilize immersion.
As a final word, We think that using this platform will help
neuroscience to generalize results to more ecological context, and in turn, will help
the Virtual Reality community to understand biological and psychological mechanisms founding
the virtual experience.
We thank you for your attention, the VRST chairs for the organization, and our co-authors
and institutions for their support.
Thank you.