Hello, my name is Deniz Arsan. I'm a 5th year Ph.D. student at the University of Illinois
Urbana-Champaign. Today, I'll be presenting "App-based Task Shortcuts for Virtual Assistants".
Although virtual assistants automate a variety of everyday tasks,
their capabilities are still dwarfed by what users can accomplish with existing mobile apps.
Therefore, virtual assistant platforms have
started leveraging apps to augment their capabilities.
Existing approaches for interfacing with apps, however, require manual development effort.
We present SAVANT, a system that automatically generates task shortcuts for virtual assistants
by mapping user tasks to relevant UI screens in apps. SAVANT uses these shortcuts to launch apps
at task-relevant screens. Here we see when the task description "send money" was given,
SAVANT automatically launches the PayPal app at its relevant screen.
In this next example, we see that SAVANT is able to perform slot-filling.
When given the task description "show me songs by Taylor Swift", SAVANT uses intent modeling to
parse the entity, and maps it to the relevant UI input. This last example shows how SAVANT
handles launching UI states that require prior input. Given the task description "order
cappuccino from Starbucks", SAVANT identifies the task-relevant screen and replays the
existing interaction trace to reach it, selecting required menu elements along the way.
Now, let's take a look at how SAVANT generates task shortcuts. SAVANT leverages data captured through
interaction mining which comprise interaction traces. Interaction traces illustrate users’
journeys within an app and encode screenshots, render-time properties and gestures for each
screen. SAVANT extracts semantic information from these interaction traces using prior work,
and computes screen representations that are used to find the best matching screens for a given
user task. A screen representation contains icons, buttons, and the text fields on the original screen.
We also found that including app descriptions from the Google Play Store provides better
matches. In addition, these representations include programmatic references - the Android Activities -
which can be used to directly launch an app at a specific UI state. There’s also a pointer to the
interaction trace leading up to the screen which is used when a screen cannot be launched directly.
SAVANT matches task queries to the semantic information contained within the representations
via keyword search. A search result contains a set of representations with their corresponding
relevance scores indicating how well the screens matched the query.
The top screen representation from the highest scoring app is called a task shortcut.
To demonstrate how task shortcuts can be used, we encapsulated SAVANT within a system-level Android
app. Android apps can restrict the activities that can be started externally from other apps; but
system-level apps can bypass this restriction. Using the activity field within the task
shortcut, SAVANT first tries to automatically launch the app at the task-relevant screen.
SAVANT is able to perform slot-filling on the task-relevant screens through a Dialogflow agent.
This agent extracts relevant intent-entity information from user utterances,
and SAVANT uses this information to perform slot-filling and interact with relevant UI components.
Directly launching the activity may sometimes fail as some activities require prior user input
to be launched because they require a specific internal state for those screens to be accessible.
When this happens, SAVANT uses the trace field in the task shortcut and replays the interactions
leading up to the component. Although SAVANT's main goal is to interface with virtual assistants and
help them with tasks they do not know how to handle, end users can also directly use SAVANT
through the app. SAVANT is not a task automation system, and it does not replicate the functionality
of existing programming-by-demonstration systems which are more akin to supervised learning.
SAVANT is an unsupervised search system; it does not generalize or automate. It merely finds the
best-matching app screen for a given task. In this way, SAVANT complements existing task automation
systems by automatically choosing the right app and launching it at a task-relevant screen.
In order to evaluate SAVANT’s performance, we conducted two user studies; one formative study
to identify common user tasks and another study to evaluate SAVANT's performance.
We first conducted a formative study with 24 participants to identify classes of
commonly performed smartphone tasks. We asked each participant to list 5 tasks
they would complete on a smartphone which resulted in a total of 120 tasks.
We grouped these tasks into 20 unique sets, and chose a representative task from each set.
We then conducted a second study with three participants to evaluate SAVANT's performance.
We bootstrapped SAVANT with the Rico dataset and computed the top-3 task shortcuts for each
representative task. We showed participants the top-3 results returned by SAVANT,
and asked them to evaluate whether or not each screen was task-relevant. We then computed top-3
precision values for the 20 task sets. The average precision over all task sets was 70.1%.
Here are some top results for selected representative tasks.
The percentage values in parentheses indicate the precision SAVANT achieved for the task.
You can also see the matched UI elements highlighted in orange.
For "message friends", you can see the "message" and "add friend" icons highlighted on this screen.
The text fields containing the exact query words and synonyms were also matched.
For "send money", we see that the relevant icon buttons and text highlighted.
"check weather" only had two matching icons on its task-relevant screen. Although this might seem
like a low number of matches for a top-scoring screen, it indicates that either the selected app
had multiple task-relevant screens or its Google Play description contained a large number of matches.
Similarly, for queries "listen to music" and "set an alarm"
we see menu items and icons being highlighted. "order food" only had two matching text fields,
which might again be explained by the additional data SAVANT considers when selecting a screen.
SAVANT performed poorly for some of the representative tasks. The top result for
“read books” was from Hotels.com, which highlights the limitation that SAVANT is currently unable
to distinguish between verbs and nouns. For “take notes”, SAVANT returned a shortcut from a
domain-specific app, myPill® Birth Control Reminder. This app allows taking notes as a side feature.
Finally, for “turn off lights”, SAVANT suggested a shortcut from the app Blue Light Filter
which allows configuring screen lighting for the night while the participants were expecting apps
for managing smart home devices. A real-world deployment of SAVANT would require strategies
for maintaining a repository of app traces, addressing potential security and privacy concerns,
and expanding to platforms beyond the Android ecosystem. For sourcing traces, relying only on
paid crowdworkers, like Rico did, for sourcing high-quality app traces would not scale. Therefore, we
envision a future ecosystem where app interaction traces could be sourced in two additional ways:
app developers could include sample interaction traces along with their apps when they update it
and interactions can be directly mined from user devices through an on-device interaction
mining app which could record users’ interactions on their smartphone as they engage with apps.
There are also security and privacy concerns SAVANT would need to address. SAVANT leverages
a system-level app because it requires two special permissions to automatically navigate
users to task-relevant UI screens. This means, for a security problem to occur, the trace SAVANT uses
for a task shortcut needs to contain malicious UI screens and interactions. This is rather unlikely
since Google Play Store already has measures to detect and prevent the installation of such apps.
In addition, SAVANT faces the risk of exposing personally identifiable information (PII) of its users,
as with any system that leverages user interaction data. One solution is to create and provide sock
puppet accounts that trace contributors can use to explore apps. However, this solution would not be
applicable for a potential on-device mining system that records interactions from real users. An ideal
solution would be to process UI screens when they are first recorded to automatically detect and
remove PII. However, current automated methods would only cover conventional PII like social security
numbers, email addresses and phone numbers. Further research is required to develop user-centered
privacy models that identify sensitive information beyond what is conventionally considered as PII.
In the future, on-device mining systems could expose interfaces where end users can redact
sensitive information from their personal traces. Lastly, in the future, SAVANT could interface with
iOS apps and apps deployed on smart devices like televisions, watches, and other appliances. Mining
and programmatically interacting with iOS apps and non-touchscreen UIs could extend this approach
even further. Thank you for listening. I would be happy to answer any questions you may have.