In this talk, I will present HelpViz, a tool for generating contextual visual mobile tutorials
from text-based instructions.
Smartphones today provide a myriad of functionalities. However, finding a certain feature can sometimes
be a challenge to users who are not familiar with the system or an app, such as enabling
tethering on the phone. So it’s common for users to look for instructions on how to perform
a specific task on the web.
Existing online tutorials are typically created by professionals or domain experts. One of
the popular formats is a web document, which presents a sequence of steps where each step
includes text instructions that are often accompanied by one or more images.
Despite existing automation techniques, the process of tutorial creation still needs a
significant amount of human effort. As software is frequently updated, tutorials need to be
revised because a task procedure might contain mismatched details or its screenshot images
may be outdated.
Therefore, it is important to further reduce the cost of tutorial creation. In this work,
we focus on utilizing the vast corpus of text instructions that are already available on
the web. We call our tool HelpViz, which can automatically transform a batch of text instructions
to contextual visual tutorials that can be interactively navigated on a mobile device.
As an overview, HelpViz transforms text instructions to graphical tutorials by extracting a sequence
of actions from each text instruction through an instruction parsing model, and executing
the extracted actions on a simulation infrastructure that manages an array of Android emulators.
The automatic execution of each instruction produces a set of graphical and structural
assets, including images, videos, and metadata such as clicked elements for each step. HelpViz
then synthesizes a tutorial by combining parsed text instructions with the generated assets,
and contextualizes the tutorial to user interaction by tracking the user's progress and highlighting
the next step.
In this demo, a user follows the tutorial to turn on the keypress sound. The user performs
the first three steps, then returns to the tutorial. HelpViz automatically scrolls to
and highlights the next step for the user. The user then proceeds with the rest of the
steps and finishes the intended task.
Two challenges arise from the automatic execution of instructions crawled from the web. One
issue is the inaccurate inference results of the parsing model, which is a pre-trained
deep model. The other issue is the mismatch between the instruction and the actual interfaces
encountered in the emulator. Both issues can cause failures for automatic execution.
To address the first issue that the parsing model produces inaccurate results, we introduce
beam search to decode the parsing model output. By default, the instruction parsing model
extracts a best sequence of operation-object tuples for actions from the text instructions
by greedy decoding, where the most likely tuple for each action is selected. However,
such greedy decoding may generate a sub-optimal
sequence if there are errors in the decoded actions, because for each step it only selects
the best action depending on previous actions, which can be a local optima considering the
full sequence. To address this issue, we use beam search to track the top 3 decoded
sequences at each step to improve the overall parsing result. Each beam is a possible action
sequence. In generating these sequences, the model aims to maximize the probability of
the overall action sequence, rather than the probability of individual actions.
The second issue is that the instruction may not exactly match the app UIs on the emulator.
For example, the instruction can be outdated as the UI of an app running on the emulator
has been recently updated. we implemented look-ahead in the executor. With look-ahead,
the executor allows a maximum of five unsuccessful actions, including tapping and scrolling,
for each task. If the executor still could not perform the action described in the current
step after five attempts, that is, the target element in the current step does not appear
on the screen, it then skips the current step and looks for the target element described
in the next step, and attempts to execute the next step’s action if found.
We conducted experiments to evaluate the performance of the HelpViz pipeline. We used the PixelHelp
dataset, which contained 187 text instructions for using the Android Pixel phones. We compare
four configurations of the pipeline: beam search only, look-ahead only, beam search
with look-ahead, and the baseline condition that uses neither beam search nor lookahead.
We looked at the percentage of actions that each pipeline configuration could complete.
Results show that by adopting both beam search and look-ahead, the pipeline achieves substantial
improvement over the baseline condition.
With the assets collected during automatic execution, HelpViz could synthesize visual
tutorials and enable step matching to serve a tutorial in context. This matching process
is done while a user is using the tutorials, and it helps to highlight the current step
for the user. With the help of the Android Accessibility API, we compare the underlying
view hierarchy of the user’s current screen with metadata stored during automatic execution
and automatically advances the tutorial to the next step.
We conducted studies with 10 participants to investigate how mobile users receive visual
tutorials generated by HelpViz in comparison with the original text instructions. Our remote
study showed that, although participants spent about the same amount of time using each technique,
participants showed overall preference for the context-aware visual tutorials generated
by HelpViz. Especially, participants found that auto-generated thumbnails and screenshots
to be helpful, and many were pleasantly surprised when the HelpViz tutorials automatically advanced
to their current steps.
We note that due to the restrictions with remote studies, 6 participants were experienced
users and would likely not significantly benefit from the improved visualizations and interactivity
of HelpViz tutorials. We hypothesize that as the complexity of the tasks increase and
users are less familiar with the tasks, HelpViz can show advantages on time performance as
well, which deserves further investigation.
In conclusion, our experiments demonstrated that the HelpViz pipeline is effective in
automatically generating tutorials by incorporating a previously trained deep model. We showed
that by enabling beam search execution and look-ahead, we improved the robustness of
fully automatic execution. The user study showed that, participants preferred the HelpViz
tutorials, even though the visual assets were automatically collected. HelpViz demonstrates
the feasibility to convert a vast number of text instructions on the web to visual tutorials
that are adapted to different versions of mobile apps and platforms and hardware configurations.