Hi, I’m Eldon Schoop and I’ll be presenting our paper “Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis”
Tapping is a core gesture in mobile user interfaces. However, because of the highly varied styles of mobile UIs,
users can have difficulty telling if UI elements are tappable.
This can harm the usability of applications, like when false affordances suggest an item is tappable when it is not
or when the design of a new feature limits its discoverability.
Designers can use tools like style guides to design good tap targets, but style guides don’t apply in all
situations, like for some games or entertainment apps.
A user study can help identify tappability issues in finished designs.
Studies produce rich feedback, but can be costly and time consuming to run.
In our work, we use a deep learning based approach to predict users’ perceived tappability of mobile UI elements.
Our work contributes to the landscape of automated UI evaluation systems, like CogTool.
However, we address some important limitations of many of these prior works, for the domain of tappability
First, collecting data on tappability perception at scale is noisy, and labels in existing datasets are not always reliable indicators
Second, existing tools can produce useful predictions of tappability, but what to do with these results is up to the judgment of designers
This is a benefit of user evaluations that’s hard to replicate with automated tools
you can get much richer feedback and even solicit ideas from partcipants for next steps.
We aim to address some of these limitations in our work.
First, we collect a new dataset that captures tappability labels from multiple raters per element.
Next, we use this dataset to train a deep neural network that predicts the perceived tappability of a region in a mobile app screenshot
Finally, we apply Explainable AI techniques to help explain the influence of the other elements in a screenshot to a specific element’s perceived tappability
And we use the latent space of our model to curate a set of relevant UI elements to to inspire design iterations
Here’s how a designer could use our trained model and its outputs.
First, you can take a screenshot of a mobile UI, and select a region of interest to predict its perceived tappability
Here, we select a menu item, shown in the magenta rectangle
The model then outputs a prediction score. Here, we see the model predicts this element is just slightly tappable (above the 50% threshold)
To better understand this prediction, we can look at our model’s other outputs.
First, we generate a heatmap showing how strongly the elements in the screenshot influence the selected region’s tappability prediction
This can help designers see when the tappability of a particular element is heavily influenced by other regions on the screen
or how introducing a new component can change the perception of surrounding elements.
To gain further design inspiration, we fetch a curated set of examples our model considers the most similar
We then split these examples based on their tappability predictions. This set of visually similar UIs with opposing influences
on tappability perception aim to help designers improve their designs.
To make this all work, we first collected a dataset of multiple crowd workers’ tappability ratings of UI elements in Android apps
We used UI elements and screenshots the RICO dataset, and selected up to 5 UI elements per screen for labeling
Each element was labeled by 5 different raters
Each worker completed up to 90 UI elements, with the median being 30
We also filtered out elements using the same methodology as Swearngin et al 2019 to reduce noise
Our dataset captured 18,667 unique UI elements from Android application screenshots
Most of these elements were typical UI features, such as text, images, buttons, and layout blocks
Notably, the majority of elements in our dataset had some level of ambiguity in their labels
and almost a quarter of them were split by a 3 to 2 vote
We are excited to share that this dataset will be publicly released on GitHub. The link is in the paper
To put this dataset to work, we created a deep neural network to predict the perceived tappability of an indicated region in a screenshot
Our model is based on ResNet-18, modified to take a full-size screenshot as input
The screenshot is concatenated with a binary mask specifying the targeted UI element
Our model outputs a binary tappability prediction (tappable or not tappable) with an associated softmax probability
Since our model only requires screenshots, it doesn’t need a fully implemented Android view, and can operate on realistic mockups
Benchmarking our model on our test set, we achieve an AUC of 0.903, reaching the state of the art.
However, we also wanted to go further than just outputting predictions alone
To highlight the most influential or “salient” regions in the input screenshot, we use the XRAI algorithm
XRAI produces a heatmap showing what regions of an input image were the most influential to a given model output
This kind of heatmap is also known as a saliency map
XRAI works by aggregating pixel-level integrated gradients into segments.
We make one key modification to XRAI.
Rather than oversample the image using superpixel segmentation, we use the native bounding boxes of mobile UI elements if they are available or can be specified
This means we can directly summarize model attributions for regions corresponding to mobile UI widgets
reducing the noise from automated segmentation methods and significantly improving the speed of the XRAI algorithm
We intend for our heatmap to help designers understand what other elements in the screen affect the tappability of the selected element.
We also want to help designers take the next steps in their design iterations by showing relevant examples based on the input screenshot
To do this, we precompute embeddings for all elements in our dataset using the latent space of our model, and use it to run k-nearest neighbors
Since the model is trained to predict tappability, the embeddings capture not only visual similarity, but also other properties of the selected UI element like as its size and location
We run KNN on a particular input, then split the nearest neighbors by the model’s tappability prediction, resulting in two sets of UIs and associated elements
which are visually similar, yet have opposing influences on perceived tappability
To filter out potentially confusing or ambiguous examples, we limit these sets to examples which have >65% prediction confidence
In our paper, we conduct an in-depth analysis of four randomly-selected elements from our dataset
and share results from an exploratory evaluation with professional designers using the six screenshots and elements shown here
Please refer to our paper for those results
To summarize, we presented a novel, automated system for predicting the tappability of mobile UI elements and explaining our model’s predictions to designers
Our work advances the state of the art by developing a deep neural network which only relies on pixels so it can be applied at earlier stages of the design process
We also use Explainable AI techniques to help generate design insights using a saliency map and nearest neighbors in our model’s latent space
We are also excited to open-source our tappability dataset where each UI element is labeled by 5 independent raters
Together, our work not only advances tappability modeling research but also demonstrates how deep learning approaches can be used for automated UI usability analysis.