Hi, and welcome to this talk about “Supercharging Trial-and-Error for Learning Complex Software Applications”. I am
Damien Masson, a PhD student at the University of Waterloo and this is work that was
done during my internship in the HCI and Visualization research group at Autodesk Research, in collaboration
with Jo Vermeulen, George Fitzmaurice and Justin Matejka.
Complex software applications such as AutoCAD, Photoshop or the 3D design software Fusion 360 that you
see here, are difficult to learn as they contain thousands of commands, and even more ways
to combine them.
While tutorials might help, many people ignore them, preferring to figure out themselves how to use
a new software. As a result, they often jump right-in and explore the software’s interface, trying
out commands as they discover them, instead of watching video tutorials or reading manuals. The issue
is that this behaviour, that I will refer to as “Trial-and-error”, is poorly supported in
software.
In this work, we contribute a framework for Trial-and-Error, including a conceptual model and a
design space to compare existing approaches and identify opportunities for future research.
As well as three techniques targeted at Trial-and-error and that we implemented in Fusion 360.
We reviewed and extracted sentences qualifying trial-and-error from eight observational and diary studies looking at
people learning new software applications.
We found several benefits of trial-and-error.
Trial-and-error is often found to perform equally, if not better than other problem solving strategies, such
as using help systems or tutorials.
Users are willing to spend more time using trial-and-error than any other approach, as they feel
like they are progressing towards their goal.
Trial-and-error happens directly in the software, saving users from the extra cognitive load and reduced space
of going back-and-forth between the software and help resources.
Trial-and-error allows serendipitous discovery of commands
However, we also found several challenges that a user might face when using trial-and-error.
A trial-and-error episode might reach a dead-end if the relevant command is hidden or disabled, or
because the affordances are poor or misleading.
Users can face difficulties understanding how to navigate a command when their mental model of the
application is incomplete or mistaken.
Even when the user is successful, the solution might be approximate, and become problematic later on.
Finally, the recovery after an unsuccessful attempt can be costly, causing accidental loss of data.
To better understand when users face these challenges, we developed a conceptual model of trial-and-error.
We identified 4 phases during a trial-and-error episode which are repeated until accomplishing the task
or reaching a dead-end.
The starting point of a trial-and-error episode is the exploration phase.
In this phase, users attempt to find commands that are likely to achieve the desired result.
Their strategy is to review labels, icons and sometimes tooltips.
If no relevant commands are found, the episode might end here. Otherwise, we move on
to the execution phase.
In the execution phase, the goal can be either to learn what the command does with
no specific task in mind, or simply achieve a specific goal. Again, the trial-and-error episode can
end here if the user cannot figure out how to operate the command.
Otherwise, we move on to the assessment phase.
In the assessment phase, users decide if they are satisfied with the result. This often boils
down to checking if the result appears correct.
Therefore, the solution could be approximate and differ from what an expert would have done.
In all cases, if the user is satisfied with the solution, then the trial-and-error episode is
successful, otherwise, we move on to the recovery phase.
In the recovery phase, users try to return to a clean state.
This can be done manually, or using undo mechanisms, knowing that the recovery can sometimes
be difficult or costly.
Otherwise, once the prior state is recovered, users can either switch to a different problem-solving strategy,
go back to the Execution phase if they want to further experiment with the command, or
return to the exploration phase to look for a different command and cycle through the phases
of trial-and-error again.
With this model, we can think about how previous work impacts trial-and-error. For example, work on
making some commands more salient mostly target the exploration phase.
Similarly, the work from Lafreniere et al. offering an option for users to swap an executed
command with another, is essentially creating a path between Recovery and Assessment.
Following this idea, we developed a design space of tools that help users be more successful
with trial-and-error.
In the first dimension, we found that users use trial-and-error to figure out four different aspects
of a software:
The user interface. For example, trying to move or hide a panel.
A command, as we showed in our previous examples.
The parameter of a command
Or, at a higher level, a workflow, which corresponds to a sequence of commands.
In our second dimension, we differentiate the moments in a which tools can intervene. Here, we
use the four phases from our model with the exception of the Assessment phase.
Finally, our third dimension differentiate the ways to present the support for trial-and-error. From simple unobstructive
modifications such as altering the interface, to opening a separate view.
Using these dimensions, we can locate previous tools targeted at software learning and better understand their
impact on trial-and-error.
Especially, we identified gaps in the literature and proposed three tools to help users during
trial-and-error episodes.
ToolTrip
ToolTrack
and ToolTaste.
Next, I will present these tools and demonstrate their use through a prototype implementation that we
developed for the 3D Design Software Fusion 360.
ToolTrack keeps track of users’ progress during trial-and-error, to prevent them from retrying irrelevant commands, or
discarding commands too early.
We do so by displaying discrete yellow triangles on all commands and parameters that have
never been explored.
We also show a progress bar on hover, to indicate how much of a specific
command has been explored.
ToolTrip informs users of possible workflows and let them go on “trips” in which all the
commands necessary to accomplish a goal are highlighted. ToolTrip provides information at a workflow-level, to show
alternative solutions and a command’s use cases.
We generated trips by mining thousands of workflows from video tutorials posted in forums.
We then augmented Fusion 360’s tooltips with ToolTrip.
Users can select among different workflows involving the command.
After selecting one of the “trips”, the commands involved will display a coloured badge and
users can look for them to guide their trial-and-error episode.
Finally, ToolTaste lets users rapidly and safely experiment with commands.
When right clicking a command, users can choose between testing the command on a copy
of the current document, or on an example specifically designed for this command.
Modifications can then be merged into the main document, or discarded, letting user decide if the
command is relevant.
In summary, we contribute a framework for Trial-and-Error, including a conceptual model and a design
space to compare existing approaches and identify opportunities for future research.
As well as three techniques targeted at Trial-and-error and that we implemented in Fusion 360.
With this research, we hope to inspire researchers to design novel techniques for trial-and-error.
Future work includes finding solutions to the aspects of trial-and-error that remain poorly supported, such
as recovering from interface modifications, or workflows.
Second, it is unclear how trial-and-error could be supported during real-time collaboration. Possible directions could be
to provide users with a divide-and-conquer approach to explore the interface, or identify users based on
their expertise.
Third, providing support for trial-and-error across applications remains an open question.
Finally, it would be useful to see how people use and integrate our three techniques in
their daily workflows
That’s the end of this talk. Thank you for your attention and feel free to check
out the research paper for more details.