We present a model that predicts visual importance across different graphic designs and natural images.
What catches your attention in a graphic design affects how you will perceive the information,
and perhaps how you will act on it.
Different design classes vary in graphical structure and distribute the attention of
the viewer across the design elements.
For collecting ground truth importance data for our model, we repurpose the ImportAnnots tool
used in prior work.
In this interface, Amazon Mechanical Turk workers mark the image regions they consider
most important to the design.
While individual annotations are subjective, averaging the annotations of 20-30 participants
generates robust heatmaps similar to human attention maps.
In total, we collected importance maps on 1000 designs, spanning 5 design classes.
We also add natural images to the mix of our training data, from the SALICON dataset commonly used
for saliency modeling.
Together, the 5 design classes along with natural images
are used to train our model both to predict the design class, and to use this class-based information
towards predicting the importance map.
Our model is an encoder-decoder style neural network that we call UMSI
Unified Model of Saliency and Importance.
Analyzing our Imp1k dataset, we find differences in importance patterns across design types.
For instance, aggregating the importance maps across all the ads in our dataset shows a
central bias for the most important content.
And our importance model, UMSI, is able to capture this same bias.
Infographics tend to have a different structure.
With titles occurring in the top left region, and being rated as the most important element overall.
And this pattern too, is captured by our automatic predictions.
And here are movie posters, the average importance map,
and the predicted map.
In all cases, UMSI learns the patterns specific
to different design classes, which it leverages in order to make more accurate predictions
per design class.
Compared to prior approaches, our model can generalize to different image types, including
natural images.
Where a state-of the art saliency model works well for such images,
an importance model trained on graphic designs does not,
but UMSI performs favorably.
On a graphic design importance dataset from prior work,
natural image saliency models can not generalize,
and while a prior importance model does well
UMSI outperforms these predictions.
Our Imp1K dataset offers a broader range of design classes
on which both prior saliency and importance models perform poorly,
but UMSI succeeds.
But importantly, because designs are a continuum, and often combine natural images with graphic design
elements and text, UMSI can adapt to the design process
to offer iterative predictions of importance that can be used within interactive tools
to offer design feedback and guidance.
As a proof-of-concept, we test UMSI within an interactive design tool.
We developed a bare-bones research prototype with minimal design support in order to evaluate
the ability of UMSI, when coupled with a genetic algorithm, to offer prescriptive feedback.
A user assembles a design on a canvas and explores the predicted importance of the
design elements.
By interacting directly with the importance score of an element, a user can specify whether
to increase or decrease the importance of the element in the final design.
A genetic algorithm operating in the back-end, receives continuous feedback from UMSI to
shuffle and resize the design elements.
The application produces new variants to reflect the user’s objectives.
While this is only a bare-bones demo, it offers a promising new direction for importance models
like UMSI to offer AI-assisted guidance within graphic design tools.
As another demonstration for importance-driven graphic design, we worked with developers
of a commercially-available design tool to integrate UMSI into a design reflow pipeline.
Users can assemble their designs and interact with individual design elements.
The design tool supports alternate layout suggestions to automate the reflowing of a
design to a new aspect ratio.
We pass the full design to our importance model in the back-end, and constrain the reflow engine
to conserve the importance ranks of the design elements.
We aim to maintain the designer’s intention: what was most important in the original design
should remain most important in the reflow.
Further details are in the paper.
We ran the generated reflow variants through a crowdsourcing task to evaluate how importance-guided
reflow using our model compares to default reflow in the existing application,
and compared to importance-guided reflow using the prior state of the art.
The design results produced by using UMSI in the back-end of the reflow algorithm outperformed
the other two approaches.
UMSI, trained on both natural images and graphic designs, ranks elements by importance
in a perceptually-meaningful way.
In this work, we have shown the generalizability of our model in predicting importance
across graphic designs and natural images, making it suitable for use within interactive
design workflows.
Our proof-of-concept demos show the promise of using importance predictions for A.I. support
within design tools, and by making both our model and dataset publicly available, we hope
to facilitate such future applications.