Hello, everyone, my name is Nataliya Kosmyna and I would like to present you our work A Pilot
Study using Covert Visuospatial Attention as an EEG-based Brain Computer Interface
to Enhance AR Interaction. This work was a collaboration with Chi-Yun Hu, Yujie Wang,
Qiuxuan Wu, Cassandra Scheirer, and Pattie Maes. We are from Fluid Interfaces at MIT Media Lab,
but also Tufts University, CMU, and Harvard. As we do have very little time today together,
I do encourage you to drop a line, in case you have any questions about this work.
So this work is gonna cover brain computer interfaces, brain sensing,
specifically non-invasive brain sensing, as well as AR interaction, specifically AR
headsets, and more specifically HoloLens. And it is a very interesting way of combining these two,
because in brain sensing specifically, we are usually suffering from the fact that
users don't really wear these types of systems, as it maybe bulky and uncomfortable. But when we have
already a system headset or helmet, specifically AR, can be also VR helmet, that users wear in
accomplishment of their tasks, we are easily bridging this gap. And we can have these sensors,
specifically non-invasive sensors, like EEG, or electroencephalography, embedded in the part of
the interaction, where one type of modality can actually elevate some of the challenges
and drawbacks of another modality. And there's a lot of interest in the state-of-the-art work
that has been already explored in the direction, I do encourage you to check the paper. A lot of this
work was focusing on using very few sensors actually, like sometimes just two or three,
and to combine them with AR headset, specifically with HoloLens. However,
there was mostly just a direct use of caps or helmets that had EEG sensors, without a lot of
effort to try and to embed the sensors and make them part of the AR infrastructure. And also a
lot of these setups, actually, we're exploring something called Reactive BCI. So BCIs that do
have some types of stimuli that would enable the interaction. For those who might know we
are talking here about SSVEPs, or P300. But I do encourage you to check the paper for more details.
So in our case, we're going to be talking about a real-time AR-EEG system based on
covert visuospatial attention in absence of any exogenous stimulation, and there is no stimulus
that would be priming the user for any action, there is nothing that is flickering, there is
nothing that would be appearing on the screen. And our contribution can be divided into sub parts.
First of all, we do have this embedded clip-on system that we have designed
for HoloLens 2 AR headset, where we have a set of the clip-on that actually sits on the forehead of
the user, and also on the back of the user. And as you can see, right now, these images,
they are becoming the part of their HoloLens 2 themselves. And they do share is a file. So in
case you do have the headset and you want to print out your own clip-on, or maybe fits your
own electrodes, it is possible. So this is how the finalized version looks like, we have this
enhancement of columns of two was a brain sensing component. On the left, you barely can see it,
it is printed in black. On the right we have printed in blue, and it's in a bit thicker print,
so you can actually see how it fits and how it fits with the HoloLens. Here's just a close up
with, in this case, a piece where you can potentially see a bit of electrodes. And
here you can see those forehead and the back of the head is a visual cortex part of this clip-on.
Then the task is actually pretty straightforward. So the user sees an empty screen, and as they
are always asked to stare at a fixation cross, that is in the middle of the screen.
And there is cue, which is a green arrow, that would be indicating the locus of attention.
There is on the each side of the screen a simple object, and the user would need to pay their
attention to this indicated object without moving their eyes. So if everything works fine that we
need to pay their attention to the right side of the screen without making movement of their eyes
to the right, one session is happening, the side object will be magnified. So we used
the HoloLens 2 built-in eye tracking functionality to collect the subjects by tracking data to ensure
that the selection of the object is not based on the eye movements, but on the brain activity of
the user all along. Here is a very, very short video to show how the actual system looks like
when it was finalized. And this is where the user is supposed to pay their attention without moving
their eyes, in this case, to the left, and right now to the right side of the screen.
Once again, please I refer you to the paper to check more in depth the results we have obtained.
And please send us an email in case you have any questions. Thank you for attention!
Transcribed by https://otter.ai