Hi everyone, I'm Chufan Lai from Peking University.
I will introduce our paper entitled
"Automatic Annotation Synchronizing with Textual
Description for Visualization".
People have all sorts of findings
in data analysis.
To share these findings, a visual presentation
would be a good choice.
That is, explain the insights while
presenting the data with visualization.
To understand the presentation, the audience
needs to match what they hear and what they see.
We call this process "text-visual matching".
It is not as easy as it seems.
Let me explain with a small example.
Now, this is a dimension reduced projection.
It depicts the traffic behaviors of over 18,000
vehicles in a national park.
I will present our findings in several ways.
First, let's pause for a moment to read.
Can you find everything mentioned in the text?
I guess it takes time, right?
What if we add some annotations?
Try to go through the description this time.
Now, how about some animations?
It's much easier to follow the presentation, isn't it?
So, from the audience perspective, animations
are better than images, and annotations are
better than pure text.
That's because, in the process of matching,
people have to switch their attention frequently.
Annotations help to highlight the targets
and bind text to image areas.
Animations help to separate multiple targets
so that we can focus on one thing at a time.
However, all benefits come with a cost.
It's easy to type in the text.
But making animations can be a mission impossible
for inexperienced users.
What we aim to achieve, is to
make it a piece of cake to everyone.
More specifically, given a visualization with
its description, we aim to automatically generate
a series of annotated animations
for a vivid presentation.
We propose a three-stage workflow.
First, we figure out what is in the visualization.
Then, we try to understand what the text describes.
At last, we match these two to understand
which entities in the image
have been described by which sentence.
It allows us to annotate correctly
and generate the animations.
Based on the workflow, we implement
a prototype system called Vis-Annotator.
It contains three modules:
Object Detection, Natural Language Processing,
and Annotation.
I'll introduce them one by one.
The Object Detection module is designed to
teach a computer to read a visualization.
More specifically, it identifies all entities
in the chart that can be visually perceived,
which we call Visual Entities.
There are two types of visual entities.
Data entities refer to geometries
that encode the actual data,
like the bars in a bar chart,
or the points in a scatterplot.
Their contours are needed for highlighting.
Auxiliary entities are those used to convey
contextual information, such as
axes, color legends, titles, etc.
We need the information they contain,
but not their contours.
There do exist classic methods
to extract visual entities.
However, they are mostly designed
for a certain type of entities.
It requires a mix of them
for comprehensive detection.
Besides, it's hard to apply them
to new visualization types.
Seeking to build a more concise and
extendable framework, we adopt
object detection techniques.
Object detection is a class of deep learning models
in computer vision.
It can identify both the type and the location
of each object in the image.
Based on that, instance segmentation models
can find out the exact image areas
and output the contours.
We use Mask-RCNN, a state-of-the-art
instance segmentation model.
An implementation of it is provided by Detectron,
an open-source deep-learning library.
Mask-RCNN is widely used in computer vision,
but never applied to visualizations.
Theoretically, it will work as long as
the target has stable visual features.
To build our own model, we first
collect a visualization image corpus from the internet.
Then we label all visual entities in each image.
After training, the model can output both
the bounding box and the contour of each entity.
However, for data entities,
the original contours are too rough
due to subtle deviations in the prediction.
You can see the irregular boundaries
in this example.
So we add an extra step to
refine the original contours,
by outlining the image area
containing the entity's major color.
With the contour, we extract other information
like color, size, location, etc.
For an auxiliary entity, we extract information
from its image area, by applying
optical character recognition
and image processing techniques.
The extracted textual information
is passed to the NLP module
to help interpret the description.
The current model is trained using
bar charts, pie charts, and scatterplots.
We get about 90% prediction accuracy
on the test set, meaning that Mask-RCNN
can be successfully applied to visualizations.
Other chart types can be supported
as long as their samples are added
to the training set.
OK, now we come to the second module:
natural language processing.
This module teaches the computer
to interpret a description.
First, let's talk about
how we describe what we see.
There are three ways in general.
You can call its name
if you happen to know it.
If you don't know the name,
you can describe its visual properties.
Or, you can describe its relationship
to another known entity.
Accordingly, there are three ways to envision
what we hear about: use its ID/type,
its visual properties, or its spatial relationships.
Our brains will generate search conditions
to help find the entity from sight.
That leads us to the task of the NLP module,
namely to extract entities mentioned
in the description, as well as their IDs/types,
properties, and relationships.
We first apply basic NLP techniques including
text segmentation, POS tagging,
and dependency parsing.
However, a word's part of speech
doesn't tell about its type or role in visualization.
The dependency tree cannot handle
different types of expressions.
We use keyword matching to
detect the types of words.
Specifically, we build a keyword library
that contains a common vocabulary
for visual descriptions.
Auxiliary information from the
Object Detection module is used to
identify chart-specific words.
We also build a structure library
to handle different dependencies
in equivalent expressions.
The key structure shows the decorating relationship
between keywords.
The above procedures turn the text into
a structured list of described entities,
which is passed to the Annotation module
as search conditions.
The NLP module can handle various
keywords and expressions, as long as
they are included in the predefined knowledgebases.
The annotation module has two tasks.
One is to match results obtained
from the previous modules, in order to
find the visual entities mentioned in each sentence.
After that, it annotates the chart
with each sentence,
and generate animations
for presentation.
The matching is done by cross-filtering visual entities
using NLP-generated conditions.
Given the contours, we highlight entities
using image masking.
The whole sentence is displayed near its target
as the annotation.
Different sentences are shown in the original order.
This demo shows how our system actually works.
You can see how the user uploads materials,
and gets a series of animations.
In order to evaluate the performance of our system,
we need to assess both the rendering
and the accuracy of the annotations.
It's also important to improve accuracy
by finding out the causes behind inaccurate cases.
First, we conduct a user study involving
10 participants, all of whom are graduate students
with computer science backgrounds.
After the introduction and training,
each participant is required to provide
2 visualization images.
It leads to a test corpus with 20 different charts.
Then the participants help assess the rendering
of a correct use case.
At last, they pick out random samples from the corpus
to test the accuracy of our system.
In the third step, participants are asked
to read a given chart with its description,
upload it to the system, and grade the annotations
considering five different aspects.
Results show that the highlighting, overlays,
and animations are doing well,
while there is still space for improvement
concerning the occlusion of annotations
and text readability.
In the fourth step, each participant is asked to
randomly pick out 12 images from the corpus.
For each image, the participant needs to
write a one-sentence description for testing.
Results are considered correct
if and only if all mentioned entities are identified
without any mistake, omission, or redundancy.
The overall accuracy is about 60%
for the 120 use cases.
We go through all inaccurate cases
to find out the causes.
It turns out most failures are caused by
NLP errors, especially the grammar mistakes
made by non-native English speakers.
Some others are due to OCR errors
while the object detection works fine.
Now, let's take a closer look at some use cases.
Here we display 6 correct cases in 3 different charts.
They demonstrate the ability of our system
to handle different types of visualizations,
multiple focuses, as well as
different expression styles.
These are two incorrect cases.
In the first case, the user tries to describe
the rightmost group of bars
but nothing is highlighted.
That's because the sentence structure he uses
is not included in our knowledgebase.
In the second case, the user also tries to
describe the rightmost group
but only one rectangle is highlighted.
That's because the name "iJustine"
in the color legend is wrongly identified,
leading to a failure in the NLP module.
So, to sum up, our system is proved effective
with about 60% accuracy.
It can be further improved by
more robust OCR and language models,
as well as more complete NLP knowledgebases.
Allowing manual correction also helps.
We also need better annotation layouts
and more prominent texts.
In the future, we plan to extend our work
to real-time speech-based annotation.
We will also try more complex visualizations
and even natural images.
Finally, I would like to thank my colleagues,
as well as funding projects and agencies.
Thank you.