Hi, my name is Ailie and I’m a research engineer at Adobe. Today I’ll be presenting
work that was done during my PhD at UC San Diego in collaboration with Adobe Research,
and this was also a demo at last year’s UIST.
People often seek help online while using complex software, such as imovie, excel, wordpress,
and many others. But the help that we find on the web lives
in a totally separate world from the software being used.
And as a result, searching for help requires us to switch our mental context, visual attention,
and input focus away from the task at hand, and onto the task of creating a search query
and browsing results. Prior work has shown that embedding help search
in the application, like Blueprint did, or using application context to improve the relevance
of online search results, like InterTwine did, helps people find what they need more
easily and more quickly And recently, our own prior work called RePlay
explored these strategies for embedding video assistance in context.
But all of these strategies are only effective if you actually search for help, and people
don’t always search when they need help. As other prior work as well as our own work
on replay has shown, people don’t always know when they are using a sub-optimal strategy
in software. And even when they do know, they might be hesitant to switch context and take
their attention off of what they're doing to search for help. Part of what makes this
context switching so difficult is having to come up with the right terminology to use
in your query that will get you the results you need.
In contrast, the ideal way to get help might be to have an expert sitting right beside
you, who knows your exact context and situation, and can give you direct assistance.
In these cases people can use language as well as gestures and their shared context
to communicate effectively with each other. And we know from prior work that language
and gesture are both extremely powerful and often complimentary to each other.
We often use both together when speaking, and one of the earliest technical examples
of this is from 40 years ago, called put that there, where here Richard Bolt uses deictic
gestures or pointing to refer to ships and locations.
"Put that" "where"
"there" Now the use of these deictic references can
be very powerful for spatial or visual tasks like this one, because it's easier to describe
things by pointing at them than by using only words.
So in this work, we wanted to see how we might bring these benefits of multimodal interaction
and input to help-seeking. And we hypothesized that moving search tasks
into a different modality might help people stay focused on their tasks while they search
for help. So to address this first question we built
ReMap, which extends our prior work on Replay, and enables 3 main multimodal features to
make help search easier. The first is using speech to initiate and
dictate a search query, the second is using speech to also navigate
the videos that come back from a search, and finally, deictic resolution, so being
able to point at interface elements to include them in your query.
So here's how the first one works using speech to initiate and dictate a search.
"the user can search for help by saying, 'search', followed by their query. 'how do I make a
text shadow.' When the user is finished speaking, ReMap initiates a search.
And second, the user can also use speech commands to follow along with the videos that come
back from their search while they work on the task itself.
"Play second video" "previous marker"
"pause" "play"
And finally, users can make deictic references by clicking on elements in the software they're
using, and referencing them with the words "this" or "that". ReMap will automatically
add those names of those elements to their search query. This can be helpful for software
that has a lot of technical domain words that might be hard to remember.
"search" "how do I use this"
So in this example, ReMap changed the word "this" to the word "noise reduction and equalizer"
when the user clicked on that tool. In order to figure out the names of the elements
that the user is clicking on, Remap uses accessibility APIs, which means that ReMap can work alongside
any desktop or web application as long as it has accessibility labels for its interface.
To incorporate speech we built a server side application that communicates with the web
speech API open in a browser window using javascript.
And to play the videos, ReMap embeds 5 video players for the search results that come back,
and our web server communicates directly with them so that the user can use speech commands
to also control video playback. To address our second research question of
whether multimodal search can help people stay focused on their task, we conducted an
exploratory think-aloud lab study with 13 participants. Participants used Canva, which
is an online graphic design software, and they were asked to choose one of these two
designs to recreate as closely as possible and were asked to use ReMap to search whenever
they needed help. We included several elements in both of these
designs that were not the most straightforward to accomplish in canva, to raise the chances
that people would need to search for help during the study.
Our main findings from this study were: people found ReMap useful for multitasking, which
meant they were able to focus on their task while searching for help at the same time.
People varied in their preferences for navigating videos, some preferring to click on them manually
while others preferred using the speech commands. And people mostly used the deictic resolution
feature for referring to elements on the canvas as opposed to tools or buttons in the interface.
Regarding multitasking, what I mean by this is participants would work on their task in
canva while searching for help and playing videos at the same time,
which is exactly what this clip I showed earlier demonstrates. Many participants had very similar
behaviour to this, following along with the videos and speaking commands when the video
got too slow or too fast. And for this, the timeline markers which are
these little green dots on the timelines of the videos were especially useful: participants
could use speech commands to go back and forth between them, and this made it a lot easier
for people to back up or fast forward to a reasonable point in the video, without having
to specify exactly how far to go. Most participants seemed to exhibit a preference
for either navigating the videos manually by clicking on them, or using the speech commands.
For those who preferred the manual way, they had the advantage that was also built into
ReMap where if you hover over these timeline markers you see a little preview of the captions.
So the people who preferred the manual way liked being able to hover over these markers
and see what they were going to be watching before they actually clicked on it.
On the other hand, people who preferred speech mentioned that they liked being able to follow
along with the video and keep it at the same pace as them, so this participant said: "If
I hear the info I want and I’m kind of following along I can pause it, take the instructions,
apply it, and then continue watching without having to stop the fluidity of work."
These varied preferences highlight the importance of providing both options in a multimodal
interface, especially as peoples’ needs might change depending on their particular
task or environment. And finally, the deictic resolution was mostly
used for referring to canvas elements. Overall, 22% of our spoken queries included
a deictic reference, and we think it's possible that deixis wasn't as useful in canva as it
might be in other more technical software, because the terminology in canva is relatively
straightforward. Most of the elements you're using are imgaes, text boxes, graphics, and
that kind of thing. So in this study people tended to use it more for convenience, referring
to things without having to say the full name. And most of the deictic references made were
for canvas elements such as images or text headers and that kind of thing. Unfortunately
only a quarter of the references people made were successfully resolved to the correct
term mainly due to missing accessibility labels in canva.
I think one thing that's exciting about this work is that we showed how you can leverage
accessibility to achieve this system-wide multimodal support, but of course this only
works as well as the accessibility labels themselves, so if application developers don't
implement labels for all of the inside elements of their applications, ReMap won't be able
to leverage that. As an example most of the references that
were unsuccessful in our study were referring to a chart or graph on the canvas and canva
doesn't have an accessibility label for charts and graphs. In general accessibility labeling
tends to be more common for the widgets, or tools or buttons in an interface, and is more
rare for the insides or the actual documents or canvas items that the user is manipulating.
Future work could address these challenges or limitations by using other methods to detect
what the user is looking at or clicking on, such as computer vision. But we also think
that a system like ReMap might work better on mobile systems where accessibility labeling
is more strictly enforced. During this study we also encountered a number
of usability challenges that could be improved in future work, many of which echo similar
findings from other work on voice search. For one, ReMap uses the web speech API to
automatically detect when the user is done speaking and then it issues the query right
away. But some users in our study would pause to think while speaking their query and the
system thought they were done, which caused frustration, as it sent the query off early.
Like many mobile voice assistants, this automatic detection of when you're done speaking can
help make the interaction smoother and more efficient. But for cases like this where users
are trying to describe a problem in more detail it might be better to have them explicitly
press a button or say a keyword to indicate that they're done and that the search should
be initiated. Second, correcting and editing mistakes in
speech queries is a known difficult problem that several participants encountered during
the study. This could be either speech recognition errors that the system got wrong or the user
deciding that they want to rephrase their query. To correct them in our study people
either switched to typing to type a correction in manually, or they repeated the entire query
over again, or in one case, a participant actually repeated themselves in real time
which resulted in a very long and redundant query. Future systems could support more natural
ways to correct oneself, much like you might do in conversation, for example by repeating
just a subphrase that you want to correct or saying something like “change this word
to that word”. Next, as the videos showed, remap displays
each word in real time while it is transcribed. While this can be useful and is often a feature
of mobile voice assistants such as Siri, in ReMap's case it sometimes distracted participants
because they were trying to keep their attention on their software to make deictic references
and refer to their task, but they instead found themselves looking at the transcription
in ReMap's interface while it was appearing. So in this case, it might be better to wait
to show the entire query once it's been fully transcribed.
Finally, as we saw, and as prior work has also shown, users tend to describe actions
they want to do in their search queries as opposed to specific tools that they want to
use. An open question for future work is how we might better support this and still help
users describe these actions using the right terminology? Perhaps by augmenting users’
queries with additional vocabulary that is related to what they’re saying could be
one way to accomplish this. So finally, here are a few things I'm hoping
you'll take away from this presentation. Multimodal help-seeking can help people stay
focused on their task while simultaneously searching for and navigating relevant help
resources. Specifically, using speech and pointing for
in-application search seems to be a promising approach for helping people communicate their
questions in a more natural way. And finally, our system ReMap introduced an
initial approach for multimodal contextual help search, that we hope future work can
build on, and we hope that our findings and recommendations that I've shared with you
today can help inform the design of future multimodal systems.
Thank you so much for listening and a big thanks to all my wonderful collaborators!