Hi, I'm Min Lu from Shenzhen university.
I'm very happy to present our work here on exploring Visual Information Flows in infographics.
This is a joint work
with researchers from China, US and Israel. Chufeng, Joel Cherry, Hanspeter, Danny and Hui.
My work is about infographics
Infographics are very eye-catching visual designs.
I'm very interested in infographics, not only because they are very visual appealing, but also
they always tell stories
By a blend of graphical elements and data in structure way.
Infographics can convey the information to readers in a nice way.
In well-designed infographics. We always know how to trace the data elements and get the story.
For example,
in this infographics, I believe most of us go through those elements in this way.
And the flow that leads us to get the story is Visual Information Flow.
In this work, we focus on analyzing this kind of flow.
Although infographics can be very differently designed, some of them do have very similar flows. For example,
in this two, the Information goes in a zigzag flow.
and in these two, the Information goes from up and down, and from left to right.
So in this work, we want to explore and learn the design patterns of visual Information flow in infographics,
so that we can have a better understanding of infographics design,
and more generally with visual organization of storytelling.
But this is a very challenging problem,
because designers may use various creative means to design infographics,
which make the analysis of their
design style super hot.
For example, the visual elements
can be designed with various appearances, even injected with the designer's personality and style.
And also the spatial arrangement of those elements can be very unique and only crafted for certain storytelling purposes.
So to do this, we collect a large amount of infographics from two designer websites, Freepik and Shuttlestock.
The infographics collected from those two websites are contributed by designers
all over the world.
And those infographics are more like design templates for the example. I show here
in these templates,
elements are always well designed. For example, icons are consistently designed, and
texts are with similar font size and style. And on the other hand,
those infographics are very diverse in their design,
which provide a good coverage of the design space of infographics.
Now, that's see how we deal with
such a large number of infographics and learn its style.
That's consider an infographics here. Designer's always creates
infographic by composing multiple layers.
And they may have a layer to deliver certain Information, another layer to create visual effects such as shading.
But basically,
an infographics can be decomposed into two layers.
A layer of artistical decorations to make the evil graphics visual appealing.
And the other layer of data elements where the information flows.
So our work consists of two parts. The first part is to peel oof those artistic decorations and get the data elements.
And then we construct Visual Information Flows from those data elements.
So in the first part, our input is the original infographics.
We leverage the power of neural network
specifically we adopt yellow and advanced object detection model
and train it from scratch with our own data set and make it able to detect the data elements.
And in the output, each element is categorized by a tag,
as well as its boundary box to show where it is. To train a successful network,
the key is to feed it with the right training data set. In our case,
we develop very simple interface to manually label the infographics. We categorize data elements
by seven groups, such as icons, texts
for those groups. We further distinguish it by subcategories.
For example, numbers, we have 10 subcategories, from zero to 10.
Using this interface, we manually labeled four thousand infographics.
And then we apply image augmentation methods, generate gray version of the infographics.
So we double the size of the training data set.
We also crop subregions to get more data set.
In the end, we get 23000 images. And with this training dataset, Twe are able to make the neural network work.
So here are more examples. The average detection precision is around 63%. It is not very very high precision,
but it's reasonable good.
There do have some miss detected elements.
And I will introduce you how we deal with those missed elements in the construction part.
But now let's look at the detection results on the left.
When we see this, we can actually have a very good guess of the flow.
It probably goes like this. And yes this is the original infographics.
So what supports us to have a good guess. Let's elaborate this problem with one more example here.
Is there anything strange?
Yes, we may notice there is something missing here.
Actually, when we read these infographics, we do some mental work unconsciously.
We compose visual elements which depict the same a piece of Information into groups like here,
icons, text and the number of years are composed as visual groups.
And we can easily get those visual groups, because designers have already adopt some design principles.
That is the Gestalt Principles in their design.
For example, they use the Proximity Principle to make elements in a group close to each other.
So we won't detect these icons are with the same group as those tests.
And second principle is Similarity Principle that groups are always configured with similar configurations here.
Each group has an icon, texts and number.
and that's why we see there is something strange when the text is missing in this group.
And another is the Regularity Principle. That is, groups are always placed in harmonious patterns.
For example, designers would avoid long jump as much as possible
to make users easily track those groups.
So with those principles, the basic idea in VIF construction is, we first take those detected elements as input,
then we construct the backbone.
And then we associate those elements in a group.
We develop a rule-based algorithm. The key idea is to scores all possible tracing of those elements. We choose the best one.
Here shows some examples.
as I mentioned before, because our neural network cannot detect
all the data elements. So in some cases, those elements can be missed,
miss detected.
So in our algorithm, we can inferred
those elements by the similarity over groups.
so we can recover them. Some of them. Here shows more examples. The grey dashed rectangles are those elements inferred.
Now we get the visual Information flow
by a backbone and a group associating links.
So to explore the design space, we transform the visual information flow to a visual Information signature.
The signature is a image patch, 100 by 100 size.
And In this image, we drove those backbones with thicker strokes.
And then we drove those group links with thinner strokes to make it a secondary feature.
So by this transformation, each infographics is transformed to a
ten thousand high dimension data.
Then we applying t-SNE projection to see it in 2D space.
All the right. I show a close up of the region highlighted in left. And we can see on the top part
the VIF are more 'W' shape. And here are most spiral and here are more like a vertical line. So in this space,
we perform the semi-automatic clustering to get the VIF patterns.
And we get six main patterns as shown here, including landscape, portrait, clock, etc.
And also another six less notable patterns.
This is the dynamic navigation in this projection
Taking VIF as a high-level feature of infographics, it also supports us to query those infographics.
We develop a prototype to show this idea.
To draw the conclusion. In this work, we explore Visual Information Flow in a large amount of infographics.
Using deep neural network to detect the data elements and construct
Visual Information Flow based on Gestalt rules. And in the end, we distill 12 main visual design patterns.
And also we discussed the taxonomy with two dimensions. But I didn't ah introduce this in my talk. If you're interested,
you can look into our paper for more details.
And we also develop a infographic searching engine, which
we can take the flow says high level features
and we also release our data set to the public.
And if you're interested, you can go to this link.
So thank you very much. Thank you for your attention.