Hi, I'm Anelise, and I'm here to tell you about a new toolbox for capturing attention
data.
Human attention provides insight into what aspects of an image are more relevant or engaging
For example--if i had a lot of data about where on an image people tended to look, I
could train an automatic cropping system that preserved the most relevant parts of the image.
If I knew what aspects of a graphic design were most important, I could do design retargeting.
If i could measure attention at multiple scales, that could give me insight into how people
explore complex images like data visualizations.
And if I knew where people tended to look at different viewing durations, that could
help me prioritize content to display or render.
Despite being so valuable, human attention has a drawback.
It has historically been difficult to collect at scale.
Say I wanted to gather attention on this image.
The typical way to do it is to bring someone into the lab and then track their eyes using
dedicated hardware.
I would measure the points on the image they looked at, and then by combining these gaze
points over many study participants and blurring them, I could create an attention heatmap.
In this heatmap, higher values correspond to regions of the image people look at more
frequently or for longer
However, this method of collecting attention DOES NOT SCALE.
Bringing subjects into lab is time-intensive, and, during a global pandemic, probably not
possible.
So in this work, we present TurkEyes, a toolbox of 4 interfaces for gathering attention data
using just a laptop or mobile phone, without eye tracking.
Instead, we make use of interaction methodologies that are correlated with attention.
With ZoomMaps, participants use the pinch-zoom gesture on a mobile phone to explore an image
at multiple scales.
With CodeCharts, participants self-report where they gazed using a grid of three-letter
codes.
ImportAnnots lets participants paint over regions of a design that they consider important.
And finally, Bubbleview requires participants to click to unblur small bubble regions of
a blurred image.
We do a deep dive on these interfaces to develop guidelines for how and when to use them to
capture attention.
Let's introduce the interfaces.
Zoom-based interfaces use a viewer’s zoom patterns as a signal of interest in regions
that are viewed or zoomed.
Previous work has explored the correlation between eye movements and the zoomable viewport
of a mobile phone, and has proposed using viewport data to measure interest in an interface
or even to create an attention heatmap over a webpage.
However, as far as we know, ZoomMaps is the first general interface for creating attention
heatmaps on images using zoom.
Here's a demonstration of how the interface works.
Participants are sent to a landing page that contains a QR code and a URL that they can
use to open an image gallery in their mobile browser.
They can then explore the images in a mobile image gallery using the familiar pan and zoom
gesture.
We can then convert the zoomed viewport data into an attention heatmap as follows:
For each pixel, we take the average "zoom level" over the viewing duration, where the
zoom level is defined as the full image area over the zoomed area.
So a pixel that is viewed up-close like pixel B will have a higher heatmap value than one
that is only viewed while zoomed-out, like pixel A.
Thus, we get a measure of where the viewer spent the most time looking in the most detail.
We can then combine the viewport data gathered from multiple individuals to obtain a zoom
map.
Moving on, self-reporting interfaces like CodeCharts show an image for a limited time
and ask the viewer to report where they were looking using a visual guide.
Our implementation of CodeCharts is based on an interface presented by Rudoy et. al.
Here is a task flow diagram for the CodeCharts interface.
First an image appears for a specific duration, usually a few seconds.
Once the image disappears, it is replaced by a codechart, a jittered grid of three-character
codes.
Once the CodeChart disappears, the subject writes down the code they remember seeing
when the image disappeared, which locates where they were looking at that time.
This video shows the interface in real time.
The codechart goes by really fast, and that's on purpose.
We want to give people enough time to read a code but not enough time that their eyes
wander.
After we've collected gaze points from multiple participants, we again convert that data into
an attention heatmap by blurring the gaze points with a gaussian, similarly to how eye
fixations are processed.
Next, nnotation interfaces allow users to explicitly segment regions that they think
are important.
One example is the LabelMe interface, used for collecting semantic segmentation labels
ImportAnnots refers to the interface of O'Donovan et al used for collecting importance annotations
on graphic designs.
Participants are presented with a series of images one at a time and asked to annotate
the most important regions.
We don't define what should be considered important
Even though there is considerable variability in the annotations obtained from individual
subjects, by averaging over many study participants, we can obtain a surprisingly consistent measure
of importance.
And this technique also works for natural images, although the segmentations might not
be as clean.
Finally, cursor-based interfaces leverage correlations between mouse movements and eye
movements, often by incentivizing the viewer to click/hover to explore points of interest.
For example the moving window methodology, like BubbleView or SALICON, reveals portions
of an otherwise-obscured image depending on where users place their cursor.
Participants are asked to explore one image at a time using their mouse cursor to click
and deblur small regions of an image.
Heatmap generation is then very similar to code charts; we blur the collected mouse points
to create a heatmap.
Now I've just thrown a lot of information at you, so might be wondering: which tool
is best? which one should I be using?
To try to answer that question, we ran experiments on Mechanical Turk.
We gathered data on a variety of stimuli types: natural images, infographics, visualizations--to
determine what insights about attention we could gather from each tool.
Here's what we found.
One thing we realized is that not all interfaces work equally well with each type of stimuli.
For example, ZoomMaps works uniquely well with large, multiscale stimuli, because of
the really natural support for panning and zooming.
While you can use ImportAnnots with natural images, it's really best suited for graphic
designs.
On the other hand, all of the images we studied support graphic designs -- as you can see
in the resume on the right.
And finally, although we did not try this in our experiments, CodeCharts can be used
for videos by replacing the image with a short video clip.
Cost is another important factor that we considered.
Tthe cost of data collection with each interface really came down to two things.
How much data does it collect per participants? and how much work is it to use the interface?
Out of the interfaces we studied, BubbleView was the cheapest.
One reason is that BubbleView gives a relatively high amount of data per viewer, as you can
see in the image on the right.
We get about 15 clicks per viewer, which means we need about 15 viewers per image to get
consistent data.
The next cheapest is ZoomMaps.
We need slightly more people, around 20, to get consistent ZoomMaps data for an image.
CodeCharts is significantly more expensive than either of the two previous interfaces,
so why is that??
Whereas with BubbleView we get many attention points per participant, for CodeCharts we
only get one.
That means that we need a lot more CodeCharts participants to get a consistent heatmap--around
50.
Finally, ImportAnnots is the most expensive interface.
Although it requires fewer people than CodeCharts, they need to do the additional work of actually
segmenting the elements in the image, which drives up the price.
We also computed similarity to eye movements by collecting data for all of our interfaces
on 35 images from the CAT2000 eye tracking dataset.
We computed the Pearson's correlation coefficient between the heatmaps generated by our interfaces
and the ground truth attention heatmaps.
We see that CodeCharts best approximates eye movements, accounting for 88% of human consistency
It's followed by BubbleView and ZoomMaps; ImportAnnots, unsurprisingly, is least similar
to eye movements.
This figure shows some qualitative similarities and differences between the interfaces.
CodeCharts best approximates human eye movements.
It shows things like exploration patterns, or single fixations resulting from exploration,
which is characteristic of eye movements.
Eye movements, CodeCharts and BubbleView fall on certain regions only, like faces, hands,
and points of content, basically salient points in the image.
ZoomMaps occasionally over-focuses on distant background elements (like the tent in the
second row) and in general is a coarser measure of attention.
ImportAnnots segments semantically important elements.
It often focuses on a single central object, and it in general highlights large portions
of the image
So what might account for differences in how well these interfaces approximate eye movements?
Our interfaces can be organized on an “intentionality” scale based on the degree to which they measure
saliency (which is more spontaneous) or importance (which is more intentional).
Saliency is a bottom-up measure of what parts of an image are most attention-grabbing, whereas
importance is a top-down measure of which elements in an image are more relevant.
Intuitively, you can think of saliency as like when you're scrolling through Instagram,
and images are popping out at you, whereas you can think of importance as more like when
you're annotating your textbook, and you're actually reflecting on the content and then
annotating it somehow.
We think that an interface’s place on the saliency-importance continuum is a function
of its “intentionality”
Eye movements capture saliency, because it requires no explicit user interaction.
CodeCharts is next because it doesn't distort the image or require user interaction while
viewing the image.
BubbleView still captures image locations that draw people's attention, but it's more
intentional bc it slows down viewing time, distorts the image, and requires people to
click to explore areas of interest.
ZoomMaps requires the user to interact with the image to gather attention data, but it
does use a familiar second-nature mechanism.
Finally, ImportAnnots measures importance instead of saliency; participants are given
ample time to consider the image before annotating anything and are asked to select regions,
not gaze points, that they find important for understanding.
These examples demonstrate some of the similarities and differences between saliency and importance
by comparing the interfaces on either end of the spectrum.
In the top row, you see an example where the two interfaces produce different outcomes.
CodeCharts is more similar to eye movements, showing center bias and exploration patterns.
On the bottom row, the central objects are more salient, so you see an allignment between
the two interfaces.
That brings us back to the question: which tool should I use?
ZoomMaps is great for multiscale content.
It has a natural interaction mechanism.
However, it is a coarser approximation of attention.
This makes it good for something like visualization debugging.
While CodeCharts can be expensive, it's a good approximation of eye tracking, especially
at precise viewing durations, which makes it ideal for an application like gradual rendering.
ImportAnnots produces clean segmentations, and it's really well-suited for graphic design
elements, which makes it ideal for graphic design retargeting.
And while BubbleView can distort the stimuli and the viewing experience, it does approximate
eye tracking well and it's cheap, which makes it great for something like training a machine
learning model to do automatic cropping.
If you have your own ideas for projects that require attention data, you can use our toolbox
to help you collect it!
The interface code, data, and demos are available on our website.
Thank you for your attention.