Conversational assistants like Siri and Alexa help users complete everyday tasks like scheduling,
home automation, and communication.
If you could similarly have conversations with Graphical User Interfaces (or GUIs in
short), what would you ask them?
And how would you frame your questions?
GUI datasets contain a rich amount of information about a wide range of applications and their
user interfaces.
For example, the RICO dataset contains UI screenshots of almost 10,000 mobile apps along
with information about their structure, content, and design.
However, finding information within these datasets can be challenging and time-consuming.
Especially for users without prior knowledge about the dataset and its APIs, or necessary
programming skills, this can be particularly difficult.
To give you an analogy, imagine having to write structured code instead of simply entering
common terms into a search engine.
In this work, we look into how natural language conversations could enable us to interact
with GUI datasets to find information about applications and their interfaces.
Using this, a designer for example could simply say "Show me some good examples of login pages"
to find relevant designs for inspiration.
To understand people's information needs, and how they would frame such queries, we
conducted an online survey with three user groups.
We considered designers, developers, and end-users as they have considerably different needs.
We shortlisted 21 key capabilities that might be useful while interacting with such datasets.
This included capabilities related to filtering and viewing specific GUIs, enquiring about
their attributes, and finding or annotating information within them.
120 participants were asked to rate each of these capabilities on their usefulness, on
a scale of 1 to 5.
These ratings were then summarised in our analysis.
A notable insight here is that some capabilities, for example, finding similar GUIs, are desirable
across groups.
Meanwhile, others are group-specific.
For example, viewing privacy info seems important only for end-users.
Our findings can inform the development of general-purpose assistants or those that are
designed for specific user groups.
Next, to understand how people might actually pose queries during conversations with GUIs,
we invited them to phrase questions to a hypothetical chatbot.
We gave them three different scenarios as prompts, to which they could respond with
up to 5 questions.
In the first scenario, a single user interface screenshot was display.
An example query here could be "does this GUI have a map view?"
in the second scenario, 5 different screens from an app were displayed resulting in queries
such as "Show me other apps from this category".
Finally, 25 screenshots from across the entire app dataset were displayed.
An example query here was "show me some shopping apps".
In total, we collected a dataset of over 1300 queries, freely formulated by participants.
We manually labelled each query according to its scope, purpose, expected response format,
and information features.
The search scope specifies whether a query refers to a particular UI, an app, or the
entire dataset.
The purpose indicates the intention of the chatbot's response – whether it is to filter
out multiple matches, find something specific, make suggestions, inform or educate the user,
or execute an action.
The response format indicates the expected type of response.
These could either be in the form of an image, text or numbers, or binary yes/no answers.
Our design space summarises the collected queries according to these three dimensions
to provide a quick overview and insights.
In addition we also identified the different information features of the dataset each query
referenced
For example app or privacy metadata, sensor usage, and accessibility information.
Our labelled dataset can serve as a useful resource for further research and development.
It can be used to identify gaps in current datasets, and could also find applications
in developing chatbots or machine learning models of users' intents.
TO demonstrate one such application of our findings, we have developed a prototype chatbot
to interact with large datasets of mobile app GUIs.
Through simple conversations.
Our chatbot can be used to retrieve information within the Enrico dataset, which is an enriched
and refined version of the Rico dataset.
In this example, the user begins with a simple question, and through a conversation, refines
the search to find appropriate results.
[Chat bubble sound effects]
Our survey material and labelled dataset are openly available for reuse.
With our work, we take the first step towards understanding how conversations with GUIs
could bridge the semantic gap for finding information in large datasets.
We are excited about the new opportunities it can provide for supporting designers, developers
and end-users during various tasks and for specific needs.
We invite you to read our paper and get in touch with us for any questions or discussions.
Thank you!
[Outro music]