hi everyone I'm Peitong and I'm an incoming PhD student at UC Berkeley I
worked on this project when I was an AI resident at Intel I collaborated with
Casimir Wierzynski and Lama Nachman who are both at Intel those of you who
designed user interfaces before know it's a very difficult process there are so many
guidelines to follow like you need to make sure your interface is intuitive
and efficient to navigate and also need to make sure it's not too cluttered
there's been a lot of prior work aimed at helping designers with this process
researchers have developed metrics and models of human performance that
designers can use to evaluate their designs as well as tools and techniques
that will optimize parts of the design like the design's layout recently there
has been a shift from using analytical models of human performance to using
neural networks these data-driven models can detect complex patterns in the data
set so they require less feature engineering the analytical models and
have been shown to make more accurate predictions for instance Li et. al. created
a model that takes in a menu and a sequence of menu items to select and
will predict how long it takes to select each item in the sequence a common way
to train neural networks to fit a data set is with gradient descent where
gradients of the loss function with respect to the model's weights can be
used to update a wish to minimize the loss function similarly if we have a
model that predicts human task performance for a given user interface
we could compute gradients with respect to aspects of the input UI like for
example its layout and then use the gradients to update the UI to optimize
for predicted human task performance we decided to explore the ability of this
technique to improve an initial UI design here's a high-level overview of
our approach first we need a model so we extended Li et al's model that predicts
menu task performance to predict a task performance metric which will be defined
later for various types of interactions on 2D mobile interfaces to
scope our work we focused on improving the interfaces layout and by layout I
mean the size and location model elements like the button and sliders
etc so for our training dataset we generated over 100 different layouts
of a single user interface and then crowdsourced human test performance data
for all the layouts next we fitted our model to this dataset and then developed
a gradient descent based algorithm that could update the layout to improve its
predicted task performance finally we applied our algorithm to improve layouts
of two distinct UIs including one that the model has never seen before and
verified improvements experimentally with crowdsourcing first some background on
modeling human performance there are many factors that affect human task
performance which is a metric we defined aims to capture how quickly and
accurately users complete the tasks the factors include the time it takes to
decide or figure out how to complete a task visual search which is the time it
takes to find the target element like the button to be tapped and then the time
it takes to perform the actual interaction like the time it takes
to tap on a button and finally learning effects where users take less time as
they become more familiar with the interface and tasks so all these
features are captured in either our model's features or architecture
so our model architecture was taken largely from deep menu but we made some
modifications so our model takes in a UI and a sequence
of tasks and predicts the task performance metric for each task we
use a hierarchical STM model and the recurrent layers help capture learning
effects as users become more familiar with the UI so on the Left we have the
encoder for each task it takes in a sequence of feature vectors one for each
element in the UI that describes the UI element and the task and at the end of the
sequence it generates embedding for a task in UI which then goes through the
predictor model that generates a scalar prediction for the task performance
metric so the entire UI is input for each task in the sequence because the UI
may change after certain interactions like for instance dragging and dropping
an element changes its location for the next task so compared to the menu model
we have an additional LSTM layer in both the encoder and predictor to
account for the increased complexity of our problem our features were selected
because they will have an effect on tasks performance
for each UI element we have 13 different features that could be divided into
these four categories we have the target feature which indicates if the element
will be interacted with during the task and next we have the elements type and the
elements type affects its saliency as well as how users will interact with it
and then we have the layout features so the size and location of each element
and our model also handles grouped elements like a group of buttons
enclosed in a container and we have the corresponding layout features for the
container and we have information on the elements label like the word2vec
embedding for a text label which will affect how quickly users can find it
then on the interaction level we have two different features the type
of interaction because different interactions take different amounts of
time and then a step feature so our model handles tasks consisting of
multiple interactions and the users have to figure out how to break a task
down into individual interactions and this is a good measure of how intuitive
the interfaces so the step feature indicates how difficult a particular
interaction is for the user to figure out for our data set to be useful for layout
optimization we have 108 distinct layouts of a single user interface and
its a photo editing UI where you can add stickers and filters to your photo and
the image on the right shows the result after a few interactions this UI has six
different types of common UI elements and includes individual elements like
this icon here as well as grouped elements like this group of buttons over
here for each layout we have crowdsourced tasks performance values for each task
in a comprehensive task sequence the sequence is the same for all layouts and
includes all possible interactions with this UI where each element is interacted
with at least five times so here are some examples of the layouts in the dataset
Out of the108 layouts in the data set five are considered good which means are
designed to meet many design guidelines then three are bad that they violate many design
guidelines to ensure comprehensive coverage of the layout space 100 layouts are
generated randomly by two different techniques we crowdsourced the task
performance data on Amazon Mechanical Turk we assign at least three workers
per layout and we have 379 participants who completed 284
tasks so we had data on over 100,000 tasks so for each task we collected the
completion time and also whether or not the worker made an error on the task so
for each task the worker is first shown instructions for the task and when
they're ready to complete the task they tap the start button and are taken to
the UI to then complete the task we aim for the task performance metric to
capture how quickly and accurately people can complete tasks so we define
it with this equation so we have the completion time for the task averaged
across all workers assigned to the layout and we threw out outliers and
this is increased by the fraction of workers who made a mistake on the task
and that's multiplied by a penalty value which is 0.8 for
major errors like accidentally canceling all your changes and 0.5 for minor
errors like accidentally adding the wrong filter we next trained and
evaluated our model on the crowdsourced data set we use the target level
R-squared metric which is also used by Li et. al. to evaluate their model and
it's defined as the R-squared of the sequence of relevant tasks for each UI
element like for instance all the times the user interacted with the slider bar
and that's averaged across all the elements so we achieved a target level
R-squared of 0.79 on our data set from 6 fold cross validation on the right we
plotted the predicted and actual task performance values for the slider bar
interactions once we have a trained model we can move on to layout
optimization we used gradient descent which iteratively updates the input to
minimize the objective function the equating for a single update is given
here so theta is the input for the nth update step and its obtained from theta at
the previous step updated by the gradient of the objective function
multiplied by the learning rate for our case theta is the layout features so the
size and location of each element as well as the corresponding container
layout features for group elements our objective function is the sum of the
predicted tasks performance values for the entire task sequence plus some
penalty functions that check for undesirable situations and will add a
large value to the objective function when they detect those situations in more
detail when we update each UI element independently we may end up with
undesirable situations like two UI elements overlapping or elements going
out the boundary of the phone screen and with differentiable penalty functions we
can steer the gradients to eliminate these situations when detected
also if designers will like a certain outcome in their optimized layout like
making sure two elements are the same size they can enforce these custom
constraints with penalty functions so we start by applying our optimization
technique to new layouts of this photo editing UI and since we're optimizing for task
performance to ensure that the output has good aesthetics we added some custom
constraints via penalty functions so the optimized layout had a predicted
performance improvement of 6.3% and to verify that this optimized layout
actually had better human performance we once again crowdsourced task performance
from amazon turk but this time we assigned at least 10 workers to each layout we
observed an improvement of 8.9 percent so the improvement in task performance
is likely due to the increase in size of certain UI elements as well as the
increased distance between some of the UI elements and also the grouping of the
sticker and the sticker buttons so we output the layout after each update
steps so one cool thing we can do is animate the optimization process so when it says
invalid like there are elements overlapping or going out of bounds and
time is the sum of the task performance values for the entire task sequence and
time plus penalty is the objective function
we also tried optimizing a layout that was initially good we first tried
optimizing with lots of custom constraints like before but the output
did not show much improvement in predicted performance and since
constraints limit exploration we decided to relax the
constraints but then the UI elements are misaligned in the output layout and some of
the elements are too small so the output layout was tweaked slightly but the tweaks
are very trivial and straightforward and just involve enlarging elements are too
small and also aligning elements to make the output look more aesthetically
pleasing so the optimization algorithm probably misaligned the elements because
misaligned elements tend to stick out more and they're hence easier to find so
the task performance will be better and experimentally we found a small
improvement in task performance but it's a smaller improvement compared to the
previous example but it's probably because it's harder to improve a layout
that's already quite good so the improvement in performance is likely
due to increased spacing between some of the UI elements
in order for our optimization algorithm to be useful has to be generalizable like designers
should not have to collect task performance data for every UI they hope
to optimize otherwise it'll just be like A/B testing but fortunately
relationships between task performance and layout attributes are mostly
universal to assess the generalizability of our model we applied our model trained
on the photo editing UI to optimize a new UI it's a recipe planning interface
where users specify ingredients they like and dislike by dragging and
dropping them to the appropriate box and when they're they'll tap the get
recipe button to see a list of recipes they may like so in order for our model to
optimize well element interaction types in this UI are also found in the photo
editing UI but the UI element types in the photo editing you are quite general and
can be used to build a large variety of user interfaces for the optimization of
this recipe planning UI we came up with a similar realistic and comprehensive
task sequence and then performed the layout optimization with the model
trained on the photo editing UI so here's an example of a initially badly
layout that was optimized using very few constraints because optimizing using
lots of constraints and not show much improvement in predicted performance so
then we had to tweak the output so we crowd-sourced task performance and
it showed a 9.2 percent improvement and this could be due to the fact that the
ingredients were moved closer to the drop targets now here's an example of the
initially good layout we crowdsourced task performance which showed a
4.9 percent improvement and this could be due to the fact that the ingredient
stickers were smaller so the drop target were closer together
also the undo icon the get recipe button were moved closer to the right to
make it more accessible to right-handed users which make up the majority of the
user base so we did pass in the fraction of left-handed users to the model so
what can we learn from all this we demonstrated that this technique is able
to update UI layouts to have improved task performance and that the model is
able to generalize and improve layouts of new UIs that it has never seen
before which means that this technique has potential to be used in practice and
as you saw in the optimization results some human input could be used to
improve the algorithm's output via specifying constraints or making minor
tweaks to the output but studies have shown that collaboration between human
and AI is encouraged over relying solely on either because both human judgment
and computer optimization can be misleading and they can fix each other's
mistakes this layout optimization system has many
uses the designer can use the model to compare the task performance of their
handcrafted layouts and they can use the optimization algorithm to generate
layout alternatives with better task performance and the system also provides a
platform for human AI collaboration where the designer and the system can work
together to develop a great layout so to summarize we extended an existing neural
network model to predict task performance of mobile UIs with a variety
of UI element types and an expanded interaction range we then developed an
algorithm to improve layouts of mobile UIs using gradients of this trained model
and also demonstrated that this model can generalize to new UI so this layout
authorization system can help designers in various ways and also provides a platform
for human AI collaboration thank you for listening