Hello everyone, I am Sarah Suleri, and I would like to welcome you all
to the paper presentation of UI Sketch: A Large-Scale Dataset of UI Element Sketches.
UISketch dataset is the first large-scale dataset of 17,979 hand-drawn sketches of 21
UI element categories collected from 967 participants, including UI/UX designers,
front-end developers, HCI and CS grad students, from 10 different countries.
Before delving into the details of this paper and what it contributes,
I would like to establish some background on this area of research.
Recent research where HCI and AI interacts, we require large-scale datasets to train AI
models. In UI design research, currently, the two large datasets we have are RICO and SWIRE. RICO
contains around 72k UI screenshots, and SWIRE contains about 3,800 LoFi screens. SWIRE was
sketched by five designers, and it is specific for the GUI similarity search system. Also,
it doesn't have annotations for training any other supervised learning models.
On the other hand, research in sketch recognition uses these other large
scale datasets. So we wanted to create a large dataset for UI lofi prototypes
and understand how humans and computers recognise UI element sketches.
We approached the research following the footsteps of prior research in
collecting drawings of real-life objects. First,
we identified a list of UI element categories that needed to be included in the dataset,
In both the paper and digital questionnaire, we show a sample UI element image on the left-hand
then we created questionnaires and collected the respective sketches from participants.
side that we randomised to get varied sketches even if the user traces it.
We used these questionnaires, and over a span of 3 years, we collected UI element sketches from 967
On the right, we provide a sketching area.
participants from 10 different countries. Our participants included UI/UX designers
with 2-14 years of prior prototyping experience, front-end developers, HCI and CS grad students.
These sketches were collected with pen and pencil using a paper questionnaire.
However in the digital questionnaire, we restricted them only to use stylus.
Once we collected all these sketches, we started our data processing. For this,
and uses contour detection to returns labelled close-cropped element sketches.
we created a script that takes the paper questionnaire as input
We then manually verified these sketches. We removed the badly drawn,
This dataset contains around 70% of digital sketches and 30%
contaminated and irrelevant sketches to finalise this dataset.
of paper sketches. We tried as much as possible to keep the classes balanced,
so we have about 850 images for each UI element category.
With this dataset, we first wanted to analyse the drawing conventions and the differences between
the sketches created by experienced designers and novices. The key difference was in the level of
details added to the sketch. Experienced designers and developers tend to keep the sketches minimal,
whereas novice designers such as the HCI and CS grad students added more detail to the sketch.
To set a baseline, we conducted another study with 76 UI/UX designers where we
Our results show that designers can identify UI element sketches with 96% accuracy.
To understand how they identify the UI elements, we looked at the UI elements
that they mostly confused. We observed that human recognition errors are usually caused
due to the confusion between semantically similar UI element categories.
identified that current state-of-the-art neural networks could identify UI element sketches with
Moving on, we trained 26 different image and sketch classification neural networks and
around 80% to 90% accuracy. The best performing model was Resnet 152 with 92% accuracy.
We further looked at how they identify these UI elements and
found that computational recognition errors are solely caused due to the confusion
between the structural similarity between UI element categories.
To further analyse the reasoning behind why the model confuses few UI element categories, we used
its ResNet152 model as a feature extractor along with t-SNE to reduce the dimensionality of the
test dataset images and ploted them as a 2D graph. This graph clearly shows that the computational
classification model clusters the images based on their structural similarities, unlike humans.
Recently a number of research projects attempted to automate lofi sketches to code.
This dataset can be used for a lot of potential applications.
As this dataset contains varied sketches,
we can generate synthetic lofi prototypes using this dataset and train AI models.
Another application would be to retrieve inspirational UI designs by sketching;
Finally, we can also create sketch autocompletion
this dataset could help train these AI models as well.
to support designers while sketching lofi prototypes.
We would like to highlight a few limitations and how we want to address them in our future work.
Currently this dataset only covers smartphone platform;
in the future, we would like to collect more sketches for other platforms as well.
Finally, as we mentioned, this dataset is provided as open-access,
Similarly, we would like to collect sketches of LoFi screens as well.
so you can find it in kaggle with this link.
I would like to thank you all for your time and attention,
please contact us if you have any questions.