Hi.
My name is Samreen Anjun, and I'm
a PhD student at the University of Texas at Austin.
I'm excited to present this book with Chi Lin and my advisor,
Dr. Danna Gurari, in which we explore
crowdsourcing strategies to track
multiple objects in videos.
We often see videos being used to track multiple objects
to study their behavior and interactions over time.
In short, this task is referred to as MOT.
And as you can see, it is valuable
for several applications, including the medical domain,
where researchers study cells to help in treating
cancers or healing wounds.
In this case, objects also split into multiple new objects
as we see in the cell video here,
and this is called lineage tracking.
These videos often contain several thousand frames,
so tracking multiple objects can be a time consuming
and complicated task.
So in recent years, crowdsourcing
has emerged as a powerful solution
as it is both cost effective and scalable for large data sets.
There has been significant development in this area,
however, two key limitations still exist.
First, most solutions rely on expert annotators,
which greatly reduces the worker pool
and limits the scalability.
And second, existing solutions have been evaluated only
on specific domains, for example, videos showing
familiar objects, such as people on cars, or unfamiliar objects,
such as biological cells.
So in this work, we focus on designing strategies,
so first, the MOT task can be as simple as possible
to engage a larger pool of crowdworkers.
Second, we wanted to examine the efficacy of these strategies
on a diverse range of videos showing
both familiar and unfamiliar content.
And finally, we developed CrowdMOT,
a new crowdsourcing system to implement these strategies.
First, we looked at two different options
to decompose this complex task into smaller microtasks.
To illustrate, we represent here a video
with six frames and three different objects.
We first analyzed the strategy recommended
by prior work in which a video is split into smaller segments,
and each segment is assigned to a different worker.
In this example, we see this video
being divided into three segments
with each color indicating a different worker.
A worker is responsible for annotating all the objects
in one small segment, which are then merged together
with other segments to produce the final video.
We call this method single segment annotation or SingSeg.
However, when we tested this approach
with non-expert workers, we noticed several problems which
we elaborated in our paper.
For example, most workers annotated only a few objects
in the video, and so the end results
had inconsistent number of annotated objects.
Instead, we proposed to simplify this task further and ask
a person to mark only one object for the entire video,
so that the annotations remain consistent.
We refer to this task as single object annotation or SingObj.
To examine these strategies, we conducted experiments
with 20 videos, of which 10 showed
people and the remaining 10 showed cells.
We used Amazon Mechanical Turk and ensured workers
had basic qualifications.
For evaluation, we used three common metrics and all
these scores ranged from 0 to 1 with one being the highest.
So the results we see for all three metrics SingObj strategy
shown here in bold produced higher quality annotations
for all videos by a large margin.
And if you dig deeper, we see it is consistently
outperforming SingSeg approach in both sets of videos showing
cells or people demonstrating its generalizability.
For our second strategy, people are
curious to examine the effect of non-iterative
versus iterative tasks.
For the non-iterative tasks, workers
were given un-annotated videos and parallelly
asked to annotate an object.
In the iterative task, workers were aloud
to see the previous workers' annotations
and build on the work together.
We performed a similar experiment
here but with a larger data set of 66 videos showing cells.
And we observed workers perform better
when they were able to see previous annotations.
We also observed that the non-iterative
round a larger chunk of videos that is shown here in green
had low quality annotations, while in the iterative round,
only two were deemed as low quality.
This further demonstrated that there
is an advantage to having workers
see previous annotations.
Finally, to implement these strategies,
we developed a new system called CrowdMOT,
which includes features to support broader
participation and videos.
We have released an initial version of an annotation
framework and we encourage you to experiment
annotating your videos with it.
Thank you.