Hi everyone, this is Jackie Yang from Stanford University
today, on behalf of my advisors Monica Lam and James Landay.
I'm here to present DoThisHere multimodal interaction to
improve cross application tasks on mobile devices. To understand
this issue better, we first look at how people use multiple apps
on their smartphones. So prior work in 2011 have
showed that 31.8% of people's usage sessions involved two or
more apps, this number may be even higher today. We conducted
a Mechanical Turk study with 75 people, asking them about what
kind of cross-app tasks they did on their smartphones, and what
apps are involved. We find eight common categories of tasks that
are usually involved in people's common cross app tasks sessions.
The four most common ones are review, which means checking
reviews of an item within a shopping app in a separate review app
password, which means insert password or two factor
authorization code from one app into another app
Compare, means comparing prices or reviews between two shopping review
apps. And finally balance, which means checking bank balance
before making any financial transactions.
However, current generation mobile OSes have limited
support for those actions. As compared to do this here.
Existing interactions are based only on voice or GUI
interactions. For example, for simple information, a user can
query information from a virtual assistant. Remember the result
and tap you back to the original app, which requires extra effort
from the user. The accuracy of the speech recognition is also a
limiting factor for this interaction. For example, in
this case, it is really hard for the user to pronounce and for
the phone to recognize the name of the restaurant correctly.
For complex information. A user can use the task switcher to switch
between recent apps, but they still have to go back and forth
between apps. To solve this. We would like to introduce DoThisHere
Including query, which allows user to get a piece of information from another app.
For example, in this case, the user tries to insert a two
factor authorization code from the duo app to this website.
User: Insert my DUO code here,
Do, which allows user to evoke another app using the
information on screen. For example, in this case, the user
tries to find a Netflix TV show in IMDb for review.
User: Find this show on IMDB.
And finally Keep, which allows user to keep a piece of
information on screen for reference, like a post-it note
while working in another app. For example, in this case, the user
tries to compare prices between Amazon and Walmart by clipping
the pricing information on screen.
User: Keep this
We build DoThisHere in a way that is portable to other
platforms. First, we have three features, query do and keep. We
build those features on top of five modules. And those five
modules are built on top of technologies that are already
offered by existing mobile platforms. For example, our
voice interface and app integration is built on top of
our in house virtual assistant Almond. But if Google or Apple
would like to adopt this in a system level, they can easily
utilize their voice interface technology and existing app
integration to make it happen. Since we're already using the
existing app integration of the virtual assistant. This also
means that app developers don't need to make a new app to
support DoThisHere. This mitigates the cold start problem. We
evaluated how well this technique works on the most common cross
tasks we collected from mturk. We find that DoThisHere can
support 95% of the cross tasks that are not well supported on
current gen mobile OS. To further evaluate the effectiveness of
our technique, we recruited 12 participants to use DoThisHere
to solve common cross app tasks and compare it with a baseline
based on Android native tasks switcher. We find that DoThisHere
can reduce user's cognitive load in multiple perspectives,
such as mental demand and effort. From the user subjective
feedback, we learn that eight out of the 12 users prefer using
DoThisHere for all tasks. Another three prefer using DoThisHere
for some tasks. So with that, we think every
smartphone in the future should be equipped with something like
do this here to help us interact with our smartphone naturally
and productively. For any questions feel free to ask me in
the following session or reach out to me by email