Hi. I'm Bruno a researcher from Inria.
I will present our work on the user preference and performance using
tagging and browsing for image labeling,
Image labeling, consists in associating images with a list of labels
describing its features on the concepts it represents.
Images are often labeled manually to categorize content and easy navigate and retrieve it.
Labels can also provide ground truth data
to train supervised machine learning algorithms.
In this scenario, they are associated to areas of the images
of video frames to provide more descriptive power.
Producing ground truth data is essential and can hardly be automated.
A report from Cognylitica Research estimates, for instance, this market
to 4.1 billion USD in 2024.
Manual labeling is also essential in scenarios that remain too challenging for automated approaches
for automated approaches to provide precise results
or give enough details based on a given ontology,
which is often the case when analyzing sport sequences.
Yet, labeling tasks remain tedious
even when splitting up the work to crowd workers.
These tasks are repetitive and last longer,
depending on the levels of details required.
Thus it remains important to study what factors impacts
the user performance to optimize it.
Labeling is usually performed via two tasks:
Tagging consists in assigning labels to a single image
and associate them to an area if necessary, while browsing consists in
focusing on the single label to assign to multiple images.
The literature focuses almost exclusively on these two tasks,
but there exist hybrid tasks combining the two approaches for which annotators
select several images to assign multiple labels to, and vice versa.
To consider all possible task variants,
we study their underlying strategies.
Let's say the annotator is facing four images that consist of nine
shapes each, and they must count the shapes for each image.
They can
assign all labels on a single image at a time, thus adopt
an image-oriented strategy, which relates to tagging a task,
or they can assign a single label to multiple images, thus adopt
a label-oriented strategy which relates to a browsing task.
We designed an abstract task to minimize the bias from expert knowledge
one might have of natural images.
Images consist of nine geometric shapes that provide three visual complexity
levels inspired from the work of Donderi, and Wolfe and Horowitz.
The lowest level relies on colors, a known pop-out feature,
and the highest on high visual redundancy.
Annotators must count
and annotate the number of each shape excluding circles that act as distractors.
There exist three label types and annotators must
select a value between two and four for each.
They must select the right labels and click on the correspondant image.
In this case, the annotator adopts an image oriented strategy.
In this second case, the annotator focuses on a label
type at a time and thus follows a label-oriented strategy.
We identified three inherent factors to labeling tasks
that likely impact the user performance, but were not investigated to date.
We vary the visual complexity and the number of images displayed at once from 1 to 9.
We also varied the label persistence by enabling buttons to remain
in the same state, thus facilitating selecting the images with the same label
or be switched off upon selecting images.
We ran two experiments.
The first focused on the strategy users adopt when given the choice.
It included 18 participants and focused on three dependent variables:
the success rates, the task, completion time and the strategy scores.
The second experiment focused on a strategy efficiency
included 24 participants and focused on the same dependent variables, excluding the strategy scores.
We defined two metrics inspired from the work of Goguey and colleagues
to assess the participant strategy.
These metrics are computed over blocks of nine images
and only for blocks involving matrices of images larger than one,
as a single image forces an image-oriented strategy.
They range from 0 to 1, 0 meaning the labeling actions
did not follow the strategy, while 1 means they all did.
We can represent the evolution of these scores
through all blocks of the experiment.
This participant had an image oriented-score
of 1 four all blocks, meaning they performed all actions on a single
image in a row and never came back to a previous image more than once.
Their label-oriented score was always close to zero, meaning
they kept switching from one label type to another.
Here is an excerpt from their performance.
In this second case,
the scores changed drastically after a turning point.
The image-oriented score averaging zero means the participant came back to images
previously visited regularly, and the label-oriented score topping at
one means they mostly focused on a single label type at a time,
as the following excerpt depicts.
We aggregated data per participant and used error bars representing 95% bootstrapped confidence intervals.
We used visual estimation techniques to estimate significant differences.
The idea is that based on two data groups, we compute the mean of the differences
between data points along with confidence intervals.
The smaller the confidence intervals are and the larger the distance is to do
the zero-line, the more evidence exists of a significant difference.
All data are available online to support replication and reproducibility.
We plotted the success rate based on each visual complexity levels.
Plotting the mean of the differences indicates strong evidence
of an impact of the visual complexity on the user performance.
Similar plots with the task completion time lead to the same conclusion.
We plotted the average scores of all blocks
in function of the button persistence and found evidence that using persistent
buttons nudge participants into using a label-oriented strategy.
We identified the user behaviors by plotting the evolution of the scores
through the experiment and running a K-means algorithm on the difference between blocks.
We found overall four behaviors.
Half of the participants used the image-oriented strategy exclusively.
One participant followed a fairly consistent strategy that evolved through time.
Three participants used an image
an image-oriented strategy, but switched transiently to the label-oriented strategy,
and the rest of the participants adapted the strategies based on the task at hand.
In the second experiment, we fixed the strategies and forced
participants to use one or the other.
For the image-oriented strategy, they had to select all label types before selecting an image.
For the label-oriented strategy,
they had to assign all images with a label type
before accessing the next.
We plotted the success rate and task completion time for both strategies.
We found small evidence that using an image-oriented stategy
strategy leads to shorter labeling times.
We looked further into possible interactions
by plotting similar data for the three visual complexity levels
and found evidence that these effects seemed impactful
for complex images.
To summarize, we found evidence that half of the participants
used an image-oriented strategy regardless of the experimental conditions,
indicating that tagging tasks are preferred overall.
Using persistent buttons seems to nudge participants
into using a label-oriented strategy as it facilitated going from one image
to the next without clicking a button again
And, an image-oriented strategy produces shorter labeling times compared to a label-oriented strategy,
especially when labeling complex images.
Based on these results, we propose three design recommendations.
Tagging tasks should be preferred when designing labeling tools.
We recommend considering the visual complexity of images when designing labeling tools
as it seems to have an impact on the user performance.
Finally, we observed more errors using persistent buttons in the first experiment, but not the second.
These were likely mode errors,
therefore designers should implement safeguards.
Thank you for your attention.