Hi, I'm Yu Zhang from the University of Oxford.
I'll be presenting "OneLabeler: A Flexible System for Building Data Labeling Tools."
This work is done in collaboration with Yun, Haidong, Bin, and Dongmei from Microsoft Research
Asia, and Siming from Fudan University.
Supervised machine learning requires labeled datasets to train models.
Different machine learning applications call for labeled datasets of various modalities
and tasks.
Repositories, such as "Paper with Code," have indexed machine learning datasets with tens
of data modalities and hundreds of label tasks.
Example data modalities include image, text, and video, and example label tasks include
classification, segmentation, and named-entity recognition.
Different scenarios require different labeling tools.
The labeling tools may accommodate specific data modalities and label tasks.
They may integrate techniques such as clustering data points and proposing tentative labels
to provide machine assistance to annotators.
The existing labeling tools usually target a specific usage scenario and are hard to
customize.
Meanwhile, developing these labeling tools is demanding for the developer as it requires
cross-disciplinary expertise.
The developer needs to know interface design, machine learning, and have solid software
development skills.
In this work, we investigate how to make labeling tool development easier.
We aim to build an authoring toolkit to ease development.
The toolkit should build on a modular composable design with unified APIs, enable reuse, and
support visual programming and static checking.
To meet the design requirements, we need to identify reusable modules in data labeling
tools.
We collected 36 flowchart figures on the human-computer interaction workflows of data labeling tools
in the literature.
The three figures in the middle are examples of them.
Then, we go through a coding process.
We group the modules and states in the flowcharts into themes and filter the themes irrelevant
to data labeling.
Through the coding process, we identified eight types of common modules, such as interactive
labeling, data object selection, and model training.
The modules are blocks in flowcharts.
We also identified five types of common states, including data objects, labels, samples, model,
and features.
The states are inputs and outputs in flowcharts.
Combining the modules and states, we get API definitions of modules in data labeling tools.
Each API can be implemented as human-computation, machine-computation, or mixed-computation
approaches.
For example, the data object selection module decides data objects' labeling priority.
It can be implemented as an active learning sampling strategy, which is a machine computation
module.
The strategy may decide to sample the data points whose labels are most uncertain as
estimated by a probability-based model.
It may also be implemented as an interactive projection, which is a human computation module.
The user can multi-select in the projection to decide data objects to be labeled.
Based on the identified common modules, we developed OneLabeler, a visual programming
toolkit, to simplify the development of data labeling tools.
Using OneLabeler, developers can develop data labeling tools by visually configuring workflows.
In the visual programming interface, through different workflow configurations, various
data labeling tools can be generated.
Here we list some data labeling tools built with OneLabeler.
They cover various data modalities, such as image, text, video, audio, point cloud, and
webpage.
They cover various label tasks, such as classification, segmentation, and temporal and textual span
tagging.
Some of them incorporate machine assistance to accelerate the labeling process, such as
providing machine-predicted default labels and using active learning strategies.
OneLabeler builds in various functions to support labeling tool development.
To provide the developer real-time feedback, OneLabeler provides a preview of the built
labeling tool.
The developer can conduct single stepping and inspect the states of the labeling tool.
OneLabeler statically checks the data labeling tool built by the developer and provides instructions
for correcting mistakes in the workflows.
The static checking saves user efforts in runtime debugging.
By iteratively resolving the issues, the developer is smoothly guided to a feasible solution.
OneLabeler also supports customization to enable developers to extend OneLabeler's representation
ability.
The customization requires coding in JavaScript.
To scaffold customization, OneLabeler's command-line interface enables the developer to start with
template code files instead of starting from scratch.
We conducted an evaluation with 8 developers.
Each participant received a 30-minute training and was instructed to carry out 4 reproduction
tasks.
The figures on the right show reference solutions for the 4 tasks.
In each reproduction task, a detailed textual specification of a data labeling tool is provided
to the participant.
Using OneLabeler, the participants developed labeling tools with, on average, around 50
minutes, compared to weeks without OneLabeler.
Meanwhile, participants' failures are caused mainly by unfamiliarity with web development.
This implies future opportunities to lower the skill barrier on building customized extensions.
In summary, we have identified common modules in data labeling tools in the literature.
Building on the common modules, we have built a visual programming toolkit called OneLabeler.
It builds in various implementations of common modules in data labeling tools and provides
development support features.
It enables developers to build customized data labeling tools efficiently with no code
or less code.
Thank you!
For more information about OneLabeler, you are welcome to visit our project website.