- So this is joint work with Forrest Wong,
Chad Franzen, and Renjita Khomar from UIUC.
Forrest is now a graduate student at Berkeley.
The other collaborators for this project were
Josh Hipsman from Northwestern University,
Dan Affergan-Yangley and Jeff Nichols from Google.
So what I'll talk about is RICO,
and RICO is a mobile app design data set
that we have recently released.
It is the largest such data set
and contains over 10,000 user interaction traces,
72,000 unique UI screens, three million UI elements,
and these are from over 9,700 popular
android apps.
This data set is publicly available
right now at interactionmining.org/RICO.
Let me show you what this dataset contains.
Okay, so if you go to the website
that I mentioned, interactionmining.org/RICO,
you will see this explore RICO button.
If you click on that it will take you
to a web app where you can visually
explore the data set.
The data set contains Android app
data from 27 different Play Store categories,
so here are, there are over 9,000 results here,
and these are the apps that we have mined.
Now let's go to one of these categories,
Maps and Navigation, and I have already
loaded that in another tab.
So here in Maps and Navigation,
we have about 270 results, and
we have organized the data set
according to categories that you can explore.
For each of these apps, we have two kinds of data.
The first is the Play Store metadata
that we show on the top.
So that contains things like star ratings
for these apps, or the number of downloads.
Then we also have one or more user interaction traces
for each of these apps.
So these traces contain a sequence
of UI screens that you can back and forth in
and get a sense of what users were
trying to do in these apps.
Here I'm also showing the user interactions
on each of these UIs with those green circles.
So this can help you understand
the different tasks that the user
is performing on this app.
For each of these UIs, the data set also,
in addition to the screenshot,
contains the Android view hierarchies.
So these view hierarchies contain
hierarchical information about
the different elements that constitute this UI.
So you can mouse over on these elements
and get a sense of where in the UI they are.
For each of these elements, we also
have important properties.
Let me show you one of them.
So we have properties that are captured here
that are, in addition to the bounding boxes,
they are the text that's being displayed
on the screen, the developer specified
class names and IDs of all these elements.
And for some of these UIs, we also
capture animations that show how that UI
responded to the shown user interaction,
and you can see those by hovering on the screens.
So let's see if this loads.
It's happening very slowly.
Okay, I'll skip this for now.
So there are animations that are there
in the data set, and if you want to
download this data set, you can go
to the Download Rico page, and we have
released all of these UI screenshots,
view hierarchies, user interactions,
and animations on this page.
And again, it's at interactionmining.org/RICO.
Okay, so that's the RICO data set,
and those are the things that it contains.
Now, to build this data set, we used
a hybrid approach for mining Android apps.
We combined human exploration with
automated exploration to reach different UIs
in all of these apps.
Human explorers can use prior knowledge
and contextual information to effortlessly
interact with a diverse set of apps.
Humans, however, tend to stick to
a few popular paths in these apps,
leading to low coverage for many apps.
Automated agents, on the other hand,
can exhaustively explore an app.
However, it is hard to develop agents
that can correctly interact with complex UIs
like the one shown here, or generate
relevant inputs across a large number of apps.
We found that combining both these approaches
enables getting higher coverage of UIs in an app,
higher than what is possible with
either approach alone.
For human exploration, we hired
13 crowd workers on the Upwork platform.
They spent nearly 2,500 hours using apps
on our system, and we ended up paying
around $2 on average for each of
these interaction traces.
Our system runs apps on Android devices in our lab.
Crowd workers use these apps remotely
over the internet through a web app
that streams the app screens to their browser.
The system captures UIs and interaction details
in real time while an app is being used.
So the overall approach here is
similar to what we showed in our project
ERICA that we presented at West last year.
However, there were two additional
operational challenges to mine data at this scale.
The first challenge was in minimizing
leaking the personally identifiable information,
of PII, for crowd workers.
And to handle that, what we did was
on our web interface we provided them
with details such as emails or phone numbers
to use with these different apps.
The second challenge was in enabling
crowd workers to complete app verification steps.
So, for example, on the Waze app there,
the crowd worker was verifying their phone number.
And to enable them to be able to do that,
what we did was we set up the system
in a way that when they use the emails
and phone numbers that we provide,
any messages sent to those show up on this interface.
So that way, crowd workers can easily
do verification cycles with many of these apps
and actually get to the actual
real screens in the app.
The other challenge was also that
getting crowd workers to follow these procedures
required some amount of training,
as well as constant oversight.
We found it easier to do that on Upwork
compared to other platforms, because
workers preferred long-term work, and
Upwork itself provided good integrated
communication tools to keep in touch
with these workers.
So that's how we ended up mining our data set.
Once we had these traces, though,
we also post-processed them to provide
two additional annotations with our data set.
First, we exposed a set of unique
UIs that we found across all of these apps,
because this is what many researchers
might care about.
This required developing a design
similarity heuristic to identify UIs
in our traces that had the same design,
and to mark them as duplicates.
This heuristic is content-agnostic,
meaning it can detect when the same UI
is presented to it with different content.
For example, the two screens shown here
look different visually, but they are
the same design.
It's just that the content is different.
Detecting such design similarity
using just pixel-based methods was hard.
What we relied on instead were
the developer specified IDs that we
captured for all of these UI elements.
These stay the same even if the content changes.
These IDs are present in the view
hierarchies that we mined, and
our heuristic compares the set of IDs
to see if two UIs are similar.
The second annotation that we provide
exposes the similarity between UIs
to enable example-based searching on this data set.
We annotate each unique UI in our data set
with a 64 dimensional vector that
concisely captures their layout.
We train a deep learning based model to
learn these vectors in such a way
that UIs with similar layouts are
close to each other in the 64 dimensional space.
So to train that model, the training
process involved two steps.
First, for each UI we create an image
that captures just its layout.
This layout image encodes the positions
and sizes of the different text
and non-text elements,
and we were able to do this precisely
without doing any image processing
or OCR because this layout information
is captured in the view hierarchy.
Then we train a deep auto-encoder
on these layout images.
This auto-encoder consists of two parts.
The first part, the encoder, shown here
with a gray background, maps the images
to 64 dimensions.
The second half, the decoder, maps it
back to an image of the original dimensions.
We train both parts together to minimize
the differences between the input image
and the reconstructed image.
Once it is trained, the 64 dimensional
encoding learned by this auto-encoder
is what we use to annotate the UIs.
Now let's look at how well this annotation
captures layout similarity between UIs.
Here I'm showing the nearest neighbors
for 6 UIs that we used as queries
to look for nearest neighbors in
the 64 dimensional space.
We see that the retrieved UIs have
layouts very similar to the query UIs.
If you look at the second and the third rows
on the left side, the model we've been able
to distinguish between the lists
that have thinner versus thicker items.
For the ones on the right side,
we see that the layouts are closely tied
to specific functionalities.
All the UIs in the first row, for example,
happen to be login UIs.
The ones in the second and third row
are galleries and date pickers.
So this is how these layout annotations
can support example based searching for UIs.
The thing to note here, though, is that
this was possible with an off-the-shelf
learning model that was trained
in a completely unsupervised manner
because we were able to accurately
construct the layout images that you see here
in red and blue, and because our data set
was large enough to meaningfully train
the auto-encoder.
So those are the things that the RICO data set
comes with right now.
Let me now talk about three things
that we are working on to further
enrich the data set.
First, we think having semantic labels
for elements and UIs would be a useful addition.
For example, we could have labels
such as this is a search icon,
or that's a login page.
We are exploring computational models
for creating such labels from
developer specified metadata such as
class names and IDs.
Second, annotating different user flows
with performance data such as completion rate or time taken
would help understand how different designs
compare in terms of design outcomes.
So we are exploring the use of
our crowd source performance testing system, ZIPT,
that I talked about earlier today,
to annotate some of the data with performance numbers.
Third, with frequent design changes being
the norm for mobile apps, we are working towards
capturing multiple versions of
an app's design over time.
Fortunately, unlike web pages, it's easy
to detect app design change for Android apps
thanks to the metadata exposed by the Play Store.
The challenge is in actually developing
sustainable ways of repeatedly crawling
a large number of apps that we can sustain
financially as well as operationally.
So those are some of the things that
we are hoping to add to the data set
in the near future.
Now let me wrap up by talking about
a few data data-driven design applications
that we hope to see built on top of RICO.
Design search systems like the ones
that we have seen in previous work for web pages
is perhaps the low-hanging fruit here.
Keyword-based search can be implemented
over the RICO data set by searching over
UI and element level metadata,
and as I've shown in the previous slides,
example-based searching can also be done
using the layout annotations that we provide.
In addition, other ways of thinking about
UI similarity outside of layout
could also be considered.
Another application is an automated generation
of UI layouts.
Automated layout generation has been
explored in prior work for graphic design
as well as for some interactive systems.
With tens of thousands of layouts that RICO contains,
this could be a useful training set
for training underlying models that power
such systems for mobile app layouts.
A related application is in generating
UI code from UI images.
This is something that developers spend
a lot of time doing once they are handed
a designer's mock-up of an interface.
Prior work has looked at ways to do this
using heuristics as well as deep learning based models.
RICO would be an ideal data set
for training such models, since it
contains both the UI screenshots
as well as details of the Android components
used to build them.
And finally, we would love to see models
that predict effectiveness of different
mobile app designs.
Researchers have collected annotations
and built models that predict aesthetics
as well as visual importance of web pages,
graphic designs, infographics,
and to some extent for mobile apps.
With tens of thousands of mobile UI images,
the RICO data set would be an ideal
starting point for collecting such annotations
and building predictive models
for mobile app designs.
In the short term, training models on
the Play Store metadata to predict
an app's success based on a design
could also be interesting.
So those are some of the applications
that we hope to see in the future.
We are also excited to see whatever else
you build with this data set.
Thank you.
(applause)
- We have time for one question.
One. Got it.
- [Audience Member] Hello, thank you very much
for the interesting talk.
I'm sure you've heard about dark patterns
in apps, on websites, that try to keep
the user on the site, and I think
these would be great databases
to write an encyclopedia or just
document different ways how this is done,
and then you can just go up, go on the
site, and look up how this app
is trying to steer your attention
and keep you locked in.
- Right. Yeah, no, that's an interesting point.
Yeah, there are so many different patterns
that are out there, and I think you were
pointing us in the direction of
can we annotate these traces to kind of
uplevel those kind of patterns and expose it to users.
And that's definitely something we want to build.
We wanna build a better search interface
around data like this where you can
go in and look for patterns, including dark interactions.
- [Audience Member] Thank you.
- That concludes the session.
We're gonna start with the closing right away,
but first, let's--