Hello my name is Lena Stubbemann.
Together with my co-authors Dominik Dürrschnabel and Robert Refflinghaus
I would like to present a new Semantic Gaze analysis approach.
Our methode relies on neural networks and is especially suitable
for eye tracking experiments in 3-dimensional
virtual, augmented and mixed reality settings.
How come, we research in this topic?
Today, virtual-reality (VR) and augmented-reality (AR) technology
is increasingly combined with eye-tracking.
Thereby, in many applications the essential question is:
“Which objects or features are participants looking at?”
Answering it however, often remains a resource-intense task.
- especially when it comes to user-specific 3D-scenes.
So far, this has been an obstacle to economic use.
Therefore, our goal was to change that.
When it comes to the state of the art,
existing approaches can be devided into 3 groups:
Manual and Bounding Shape Approaches,
Tracker and Model-Driven Approaches,
and Computer Vision and Data Science Approaches,
which our approach can be accounted to.
In this group, we see the highest potential to
develope an „one-fits-all-solution“ over the whole XR range.
However, one main limitation, which comes with these apporaches
is the availabilty of suitable annotated data bases.
This is especially critical, when non-stand-alone Volumes of Interest (VOIs) are to be annotated,
This is, because the granularity of pre-annotated data bases
is often restricted to delimited objects, such as cars or trees,
and does not suit annotations on a feature level, such as the objects‘ dampers, levers or buttons.
To overcome this problem, our approach includes a methode
to built suitable data bases from scratch almost fully automatically.
Therefore, our main contributions are as follows:
First: We show how arbitrary new training data sets for the annotation of VOIs
can be built up almost fully automatically doing image augmentations with Cycle-GAN.
Second: We present a machine learning approach to
automatically annotate VOIs at a feature level
and only on the basis of synthetically generated training data.
To do so, our approach consists of 4 main steps,
which I am going to explain now.
In the first step, we use a virtual model to prepare training data.
To do so, a marker is created and systematically moved across the surface of each VOI.
This is done by defining a moving path for the marker.
The resulting simulation is then exported as frames
and cut to thumbnails around the position of the fixation marker,
as can be seen here.
To make those training data suitable to the actual experimental scene,
while avoiding intense simulation efforts,
we then perform an image augmentation.
Thereby, we perform a channel-shift to transform the simulation data
into images,that appear to look like the desired target domain.
To built a target domain, we again add a marker,
but this time to experimental data frames
and again cut the frames to thumbnail-size.
Here, you can see, how the channel-shift is performed.
To do so, we use a generative adversarial networks, called Cycle-GAN.
This architecture can be used to transfer images,
between two different domains, in the absence of paired examples.
In our case Cycle-GAN translates an image
from the domain of the simulation data (source domain),
to the domain of the artificial fixation data (target domain).
The performed channel-shift can be seen here,
and goes from A to B.
Performing the channel-shift,
we now have a suitable synthetic set of training data,
which we can next use to train our classifier upon.
For this, we use a convolutional neural network called ResNet.
At last, we can now use the so trained classifier,
to predict the VOIs of our experimental data.
We do so, by classifiying every frame of the gaze replay,
which containes a fixation.
We then perform a popular vote
to determin the overall VOI result.
To evaluate our approach, we tested our methode
in two experimental design studies on coffee machines;
one being a real-world setting, the other a virtual-reality setting.
VOI annotation was done on a feature level, with 10 different VOIs,
which can be seen here.
As baseline we used EyeSee3D a geometrical model-driven approach.
As ground truth results, we relied on manual annotations.
When it comes to the tracking apparatus,
we used SMI mobile eye-tracking glasses.
For fixation detection and the creation of gaze replays we used BeGaze 3.7.
To fullfill the technical demands of our baseline EyeSee3D
we furthermore needed a SMI 3D-6D head tracking
and an outsinde-in motion tracking system, which was OptiTrack Primex 13W.
When it comes to network training,
we used a thumbnail size of 224 x 224 pixels.
For image augmentation we used the Cycle-GAN architecture, as proposed by Zhu
and a trainings data set of 1,000 images for each domain.
When it comes to CNN classification, we used a ResNet50v2 architecture as from He
with an output layer of 12 neurons, which is 1 neuron per VOI and 2 default classes.
Training date here were 100,000 artificial images for each domain.
As training parameter we used an adam-optimizer, a learning rate of 0.001
over 20 epochs, and a sparse categorical crossentropy loss.
The results of the experiments can be seen here.
As you can see:
Overall, our approach performs slightly better in virtual reality than in the real world.
This is expectable, as the simulation data resemble the virtual-reality optics better.
Still, our approach can compete with EyeSee3D,
as both approaches reach comparable results, as can be seen in this comparison.
Looking into the proposed process in detail, we furthermore asked ourselves:
First: Is it reasonable to use simulation data instead of pre-experimental data
as a training basis? And second:
How big is the performance share of the image augmentation?
To answer the first, we took 30,000 pre-experimental images,
which then need to be annotated manually,
and trained a new ResNet classifier upon it.
The results can be seen in this line.
While the generation of the data base took around twice the time of our approach,
the results show, that the data base was still not broad and precise enough
to train a complex neural network such as ResNet50v2.
Our approach can thus be seen as economically more viable,
as it achieves a much higher accuracy at lower time input.
When it comes to the performance share of Cycle-GAN,
we trained a ResNet architecture exclusively on the un-augmented simulation data.
The results can be seen in this line.
As can be seen, Cyle-GAN is boosting the accuracy tremendously
and this is true for both setting.
Using the image augmentation seems to compensate the reality deficits
of pure simulation data sufficiently.
Overall, our work should therefore be seen as a proof of concept
for a new method to annotate VOIs in interactive 3D-settings,
which reaches state-of-the-art accuracy, while allowing cross-platform use.
For a deeper discussion please refer to the paper.
Thank you for your attention.