I’m Sara Bunian and I’ll presenting the paper VINS Visual search for mobile user interface design,
and I’d like to take to thank my co-authors for their help making this work possible.
Visual search is about finding design examples supporting the design requirements.
However, designers usually struggle to articulate their design requirements into keywords.
The reason is that designers describe these requirements as
images across the different design stages. So, it would be more suitable to use these
UI images, instead of keywords, as input queries for finding relative design examples.
And that’s the main research question we try to address in this paper.
To answer the question, we conducted semi-structured interviews with 24 designers
to gain a professional perspective into the search process in terms of different aspects.
First, most designers agreed that the examples are important and are widely used within the projects.
Second: designers identified difficulties which mostly relates to describing the specific
UI with only text queries. Third, instead of only keywords,
designers pointed that other design aspects should be considered
when measuring the UI similarity such as structure and functionality.
This emphasizes the need to consider new design aspects
when developing effective design searching tools. Taking these design aspects into consideration,
we advance the current research by first modeling the structure and content of the UI instead of
treating it as only a pixel-based image. Second, supporting UI queries of multiple design stages.
This is our proposed retrieval pipeline. It starts by using an object detection model to detect the
labels and bounding boxes of the different UI components in the input query. This will provide
a structural representation of the layout. For this we have collected and annotated a
large-scale mobile UI dataset of different design stages.
The system, then, creates a segmented image and pass it to a multi-modal embedding network that
uses attention mechanism for retrieval. We collected our dataset because we need
high-quality annotations for training the object detection which is not
currently available in the public UI datasets. Examples of these limitations are as
shown in the Rico column First, Broken hierarchies
Second, Extra space within the bounding box and Inconsistencies with the
class label of similar objects As a result, the first contribution
is the VINS dataset which has a total of 4,800 annotated UI images across two design stages
The second contribution is the attention-aware retrieval system,
The image model encodes the hierarchical structure of the input segmented image
by using a box attention mechanism. The label model encodes the UI content
by representing the UI classes. The learned embeddings from the two models
are concatenated for the retrieval part. More details are available in the paper
We first evaluate the annotations of VINS dataset with object detection. Overall, VINS
annotations provides a 26% increase over the mean Average Precision of Rico dataset annotations.
Next, we evaluate on the complete VINS dataset, across 12 classes
and achieve an overall mean Average Precision of 76% and Area Under the Curve of 79%.
Second, we evaluate the retrieval system by calculating the precision score at top K
retrieved images , Overall, our model
outperforms the baseline and further improves the precision rates at each value of K.
Finally, we evaluate the end-to-end combined model on input queries belonging
to different design categories and stages. We recruited 5 designers for this evaluation.
These are high-fidelity queries along with top-3 retrieved results.
All the designers mentioned that all the retrieved results are relevant to the input queries and
would provide beneficial design examples. Theses are results for Wireframe UI queries.
Designers were also satisfied with them. As an extra feature, the model can also be used
to auto complete UI layouts when giving partial design requirements
as shown in the figure. Thank you and if you have any questions