we present user x machina an
investigation
into simulation as a design probe in
human in the loop
text analytics topic modeling is a
common approach to the analysis of
textual data
a bunch of text documents are provided
as input to a machine learning model
like lda
which derives a set of topics that are
then outputted for a human to review
now consider the following scenario a
person reviewing this
output wishes to break up the larger
topic into smaller subgroups
and also to try to understand what the
impact of this change could be
on the other documents the person can
take what they've learned reviewing the
model output
to modify the parameters of the lda
algorithm the topic model then produces
a new output for the person to review
now maybe the person was expecting just
a minor refinement
a breakup of that one topic and a
rearrangement of some of the smaller
ones
instead the topic model produces
something unexpected
perhaps because of an overcorrection or
because of the way that it's
incorporating user actions into its
refinement
overall the process can be really
disorienting and confusing to a person
this example and others like them led us
to ask the question
of exactly how much do user interactions
impact the refinement of the topic model
and most importantly the visualizations
that we use
to show the results we especially wanted
to understand the relationship between
trivial and significant user changes to
a topic model and the resulting data
visualization
for example if the user wanted to make a
smaller trivial change
there should not be a great deal of
change in the outputted data
visualization
alternatively if they make a larger and
more significant change
this is something that the data
visualization should reflect
and of course we wanted to know of
scenarios where the magnitude of the
user changed
does not match the change in the data
visualization
our approach to explore this problem was
the conducted human-centered
sensitivity analysis of topic modeling
our approach has several steps first we
looked at prior research on topic
modeling
and looked at the types of interactions
that people wanted to do to refine a
topic model
for example to merge or split clusters
or to add new stop terms
next we designed a pipeline around lda
based
topic modeling our topic modeling
pipeline had four stages
an input stage to take in the data a
preparation stage
a modeling stage that ran the lda
algorithm and finally the assessment
stage which computed a set
of benchmark cluster and topic metrics
next we simulated various types of
actions that a user could take
across the different stages of the
pipeline for example
adding removing stop terms or documents
changing the number of clusters
or even merging and splitting clusters
we had a baseline run where the lda
model parameters were initially
optimized to the data set we then
introduced a number of simulated user
actions
and measured the impacts of these
actions again with
benchmark cluster and topic metrics
in this figure the results of the
baseline run are shown in red
and we show the effects of user
interactions as box plots that
deviate from this red line
next we visualize the results from the
different user
interactions again relative to a
baseline run
we used a pie chart a topic term matrix
as well as
ranking tables to show the results of
the lda model
we compare these visualizations relative
to our prior numerical summaries
of how much different user actions
impacted
the model output our primary finding was
that different types of user actions
impacted the topic model differently in
particular
those changes earlier in the process in
the preparation stage appear to have the
largest impact
on the model's output our second finding
was that data visualizations convey the
effective user actions
differently here we compare the baseline
run
against several simulated user actions
having low to
a large impact on the model output
we compare how these are shown using the
pie chart
the topic term matrix as well as the
ranking tables
overall our findings show that user
actions
are not created equal when interacting
with topic models
we call on visualization researchers to
surface the impact of actions
to users to alert them of potentially
disruptive changes
and to help guide them through the
parameter space so that they can decide
what to do next you can find our paper
and all the data supporting our analysis
at these web links