In this video, I present our research on
Designing
Interactive Transfer Learning Tools for
Machine Learning Non-experts
Interactive machine learning or IML help
users to integrate their knowledge when
building complex machine learning models.
While a number of IML tools exist that
are geared towards expert users,
less is known about how IML tools can
support non-experts.
Yet non-experts have a lot to gain from
machine learning technology.
For instance, an online community
moderator might want to use
ML to help them better manage negative
posts.
In this paper we explore how Interactive
Transfer Learning
might support non-experts.
Transfer Learning is a popular machine
learning approach in which a model
trained on one task can be adopted for
other related tasks
using a smaller training data set.
However transfer learning is primarily
an expert tool because it requires a
series of interdependent subtasks such
as
identifying transfer candidates
transferring them successfully
and identifying subsequent steps.
To inform our design, we conducted
semi-structured interviews and a
literature survey
to identify salient design requirements
synthesizing them into a series
of tasks related to selection assembly
and diagnostic
this synthesis allowed us to think from
the perspective of non-experts and
identify key
functional and conceptual challenges in
acquiring a clear sense of how to engage
in transfers.
Following our inquiry we implemented a
prototype tool called InteracTL
which helps to scaffold the complexities
of transfer learning in Convolutional
Neural Networks
and makes them accessible to ML non experts.
IntracTL
uses a building blocks metaphor to help
users assemble
train and test CNN models.
Using expert curated pre-trained models
users can interactively create a model.
Users can configure samples from classes
they want to train.
The users can test the trained models
for performance on specific samples
assessing the final accuracy. Using
pre-existing samples from the test data
set users can analyze
model performance by probing with
different inputs.
InteracTL visualizes the model's
learned representations to provide a
sense of model architecture.
And finally, InteracTL provides a
comprehensive view of the model
performance through
pan and zoom scatter plot and confusion
matrix.
Post-implementation, we conducted a user
study to explore how non-experts made
use of InteracTL
Our study involved 15 participants
between 18 to 25 years of age
all with a non-computer science
background. Participants were given the
task to design a model that can classify,
as accurately as possible, handwritten
samples of lowercase alphabetical
character.
During the study participants could move
through different stages of model
building
as they required and build models using
representations
from related tasks. For instance a user
might create a hybrid model by
transferring learned representations
from model trained on uppercase
characters and empty computational units.
The users could build and iterate over
as many models as they preferred
We use state diagrams to capture the
interaction patterns of each user as
they move through these steps.
We observe three different patterns of
activity among users.
Some users focussed on the data, others on
the model and some more
on the balance between the two. For
participants who focus on the model
their transfer decisions were motivated
by domain knowledge and task
similarities.
They used What-If Analysis to study the
model's current state
to integrate their knowledge.
Participants engaged in hypothesizing
about letter shapes and their impact to
inform
building several different models over
multiple iterations.
For designers of interactive ML
systems it is critical to consider how
to offer tools that reduce the cost of
exploration
and provide precise implications of
altered model architectures.
We observe participants relied on their
perceptions of statistical learning
process
resulting in unsuccessful transfers.
Comments like
"I want to see if it remembered anything
from last time", demonstrate that their
understanding of the machine learning
process is grounded in human learning
processes
For designers of interactive systems,
these misinterpretations may be
mitigated by
using visualizations of the declarative
knowledge of the model clearly
communicating its learning style.
Our observations also show that
participants over relied on accuracy
number
and struggled to find patterns across
correct and incorrect model outcomes.
Participants comments demonstrate that
they often
rebuild the models to retrieve
information across transfers.
For designers of interactive systems, it
is crucial to surface historical and
process data
that aligns with their hypothesis and
evaluations.
We invite you to check out the full
paper. Thank you.