Image-based projection labeling When recording mobile eye tracking data, we
face the problem that each video is individual and comparisons between participants become
difficult.
This example shows 3 different recordings of a person looking at magazine covers.
Although the same content is visible, it is difficult to compare the viewing behavior
just by watching the videos with a gaze replay.
In such cases, areas of interest (AOIs) are included to make the recordings comparable.
With this semantic information, we can, for example, calculate dwell times on AOIs, or
perform scanpath analysis.
How can we get this information about AOIs?
One established method is the use of optical markers.
Depending on the environment and type of AOIs, this is not always possible.
Often, manual annotation is necessary.
To do this, polygons can be drawn on AOIs and have to be tracked through the video.
This step has to be repeated for each object in every individual recording.
Manual annotation is very time consuming but often the only way to analyze the data with
AOIs.
As an alternative, object recognition can be applied to detect AOIs automatically.
However, this requires that the algorithm is trained for the objects that will appear.
With image-based projection labeling, we present a new approach to label data from multiple
recordings simultaneously.
And the annotated data is directly visualized for the semantic interpretation of scanpaths.
The approach is based on the idea to represent each video by as few images as possible which
represent dwells on AOIs.
By extracting a thumbnail for each sampled gaze point, we will get very similar images
as long as a person was fixating something.
For the labeling task, this sequence contains too much redundant information, because all
of the similar images belong to the same AOIs.
Consequently, we can segment our data into periods with very similar images.
To perform a segmentation of the data, we calculate the similarity of thumbnails in
consecutive frames.
This plot shows peaks when the similarity between the thumbnails decreases as an indicator
that the point of regard switched to a visually different area.
For better interpretation of the plot, we can visualize the content with vertical scanlines
through the center of each thumbnail, which result in a slitscan visualization.
Superimposed with the plot, this visualization helps to find a threshold for the segmentation.
When a person was fixating the same area over time, the slitscans show a repeating pattern
and also abrupt changes can be identified easily.
The threshold for the segmentation can be set for each video with a live preview of
the resulting segmentation.
Low values result in smaller segments, down to individual frames.
Each segment is then represented by a thumbnail and the duration of the segment is indicated
by a drop shadow.
So far, we have a collection of images that we now have to investigate in a convenient
way to identify AOIs and assign the correct labels.
Hence, the next step is the projection of the segment thumbnails.
Each image can be represented by an n-dimensional feature vector which can be projected into
a two-dimensional space that we can visualize to search for gazes on AOIs.
For dimensionality reduction, we use UMAP because it performs fast and results in visual
clusters of similar thumbnails, which mainly contain the segments relevant for the AOIs.
To control the elements in a visual cluster, selected thumbnails can be rearranged into
a grid.
Small previews can be displayed to see the thumbnails in context of the original video.
When we are sure that all elements belong to the same AOI, we select them and assign
the label.
In this example, all thumbnails belong to AOI 3.
This AOI information becomes also immediately visible in the timelines that show the segmentation.
We can repeat this procedure for all regions where we identify visually similar elements.
Thumbnails that do not belong to the AOI we are currently looking at can be identified
and removed.
Here for example, we have a thumbnail that still belongs to AOI 3.
If we cleaned the cluster, we can assign the correct label.
In this case, it is AOI 1.
It is also possible to investigate segments starting from the timeline.
Clicking on a segment shows the corresponding video frame and highlights the element in
the projection.
This way, outliers in the projection can be identified quickly, as they often appear as
unlabeled segments between two identical labels.
With this approach, we can label the three videos simultaneously in 5 minutes.
The manual annotation of the same dataset with polygons took about 2 hours.
In the paper, we show different use cases where we applied this technique.
One of them is the personal encounter dataset from one of our previous publications.
It contains the recordings of three coffee breaks with personal encounters of a person
wearing eye tracking glasses.
The projection not only shows clusters of the different persons, it also separates the
data recorded outside and inside of the building.
Clusters containing mainly the thumbnails of specific persons are highlighted by color.
This example shows that even without a sophisticated face recognition algorithm, image-based projection
labeling can be applied to annotate gaze data from personal encounters.
I presented the concept of image-based projection labeling as a method to rapidly annotate AOIs
from multiple recordings simultaneously.
With each annotation, the results become immediately visible in the visualization and can be interpreted
in their temporal order.
For future work, this approach has the potential to serve as a basis to provide AOI labels
for the training of machine learning algorithms.
We plan to extend this idea in this direction to create a user-centered active learning
process.
Overall, image-based projection labeling provides a fast and easy-to-interpret way to make data
from mobile eye tracking experiments comparable.