- All right, so yeah, my name's Alex Kale,
I'm a second year PhD student in the Information School
at the University of Washington,
and I'm gonna present on some work that I did with
my advisor, Jessica Hullman,
who's at Northwestern University,
and Matthew Kay at University of Michigan.
So researchers often make hard choices about how to
collect, analyze, and present information.
Together, let's consider the competing goals and incentives
that influence this decision-making.
Can I get a show of hands,
how many of you have ever set out to
write a related work section for your paper with
aspirations to thoroughly summarize
all of the critical prior works related to your project?
Should be most of us, right?
Now, how many of you have ever failed to achieve
that great related work section that you set out
to write, because of constraints on your time
leading up to a deadline, or constraints
on the available space in a paper?
Right, so we argue that researcher decisions are
characterized by such tensions, between idealized goals
and constraints in practice.
Often the optimal choice of analysis is unclear,
and the researcher must develop a rationale
for their decisions, sometimes principled,
and other times arbitrary.
Prior work refers to instances where researchers
use their discretion as researcher degrees of freedom.
And although this flexibility in analytical decision
is often framed as an issue of methods or incentives,
in our study we investigate it as a design problem.
We interview researchers to better understand
their decision-making practices,
and we ask how we can design interactive systems
to support researchers in making these analytical decisions.
To investigate researcher decision-making,
we focus specifically on the context of research synthesis.
Research synthesis often entails a systematic review
of the scientific literature on a topic,
that is a structured process for defining a question,
querying the literature,
and coding each article in a review.
Sometimes, in addition to systematic review,
research synthesis will involve quantitative aggregation
of evidence through meta-analysis, that is the procedure
of collating results across studies,
and combining them into an overall estimate of effects size,
through a statistical model.
By effects size, we mean the standardized quantitative
estimate of the impact of some interventions,
say, mindfulness therapy, on an outcome of interests,
say, incidents of diagnosed mental health disorders.
The end products of research synthesis are a written report
of findings and process details,
as well as a recommended course of action,
especially in cases where research synthesis is conducted
within an organization in order to
support evidence-based decision-making.
I wanna point out that research synthesis is distinct
from research in general, in that it mostly involves
a recombination of prior work
to address a practical question,
rather than a brand new novel study.
So to investigate research decision-making practices,
we conducted open-ended conversational interviews.
The interviews were designed to get researchers
to discuss their analytical decisions through
specific examples.
We used a combination of snowball and convenience sampling
to recruit interviewees who regularly use systematic review
and meta-analysis to solve practical problems
within their respective organizations.
We recruited 11 professional researchers in total,
that's four professors and one postdoc
at the University of Washington Medical Center,
who study a variety of topics,
two PhDs at the Seattle Veterans Affairs Medical Center,
who study treatments for post-traumatic stress
and substance use disorders,
and four scientists at the United States Naval Air Warfare
Center Training Systems Division,
who evaluate various trainings for soldiers.
Now, we don't claim that this sample is
completely representative, but we do think that it reflects
the high variability among applications
for research synthesis.
The interview guide was a list of topics of interest,
and we had first broached each topic by asking,
"Did you ever, for example, exclude studies
that you considered in a review from a meta-analysis?"
We would then ask them to describe a specific example
in order to elicit contextualized descriptions
of research synthesis practices in retrospect.
We would then use these concrete examples they provided
to ground further discussion about how they made decisions,
and we grouped the topics of interest in the interview guide
by the stages of the research process
that we discussed earlier.
In our analysis, we identified 480 decisions
that interviewees mentioned.
We also identified 21 different reasons
that researchers give for their decisions,
and in subsequent analysis we grouped these reasons
into six motivating factors that
drive researcher decision-making.
To help us consider the design of interactive tools
for research synthesis, we kept track of user needs
around decision-making, especially when users
explicitly stated the need for improvements
over the tools they currently use.
Now, all these decisions, reasons, and needs,
emerged through open coding.
We also integrated theories from related prior work
to deepen our analysis of researcher decision-making.
To understand our decision analysis,
envision the analysis process as a series
of decision points, each potentially influencing
subsequent choices and analytical results.
To borrow a metaphor coined by the statistician
Andrew Gelman, we can think of the research process
as a garden of forking paths, where each decision
is like a fork in the road.
So, how do researchers choose which path to take?
Thankfully, we can borrow from our review
of a large body of literature on judgment
and decision-making, conducted by Lipshitz and Strauss.
They identify three broad strategies that people use
when making decisions under uncertainty,
acknowledge, reduce, and suppress.
The first strategy researchers could use
is to acknowledge uncertainty, and they might do this by
exploring alternative analyses,
and thinking about how to preempt problems
that they might encounter later on.
For example, consider this quote from a researcher,
we'll call him Joe.
"When the search term for database query
in a literature review isn't clear, it's a nightmare.
It's this sort of ongoing iterative process
of trying out different terms,
and the conversations with collaborators
have been the best way to narrow the scope
of the search in my experience."
The behavior described in this quote is illustrative
of two themes arounds the use of acknowledgement.
One, that researchers explore their options,
because they're motivated to perform
a comprehensive analysis.
And two, that rationales for planning decisions
often emerge through interpersonal communications.
If these rationales are documented at all,
it is often in the form of informal note-taking.
We are interested in how interactive systems
could more explicitly represent this iterative search
over a set of queries, and the social process
by which decisions are made.
In our interviews, we see acknowledgement not only as
scoping early on in the research process,
but also later on.
For example, when interviewees describe using caveats
to communicate with stakeholders about potential limitations
of their analysis.
Acknowledgement is an important part of
planning and communicating, but in order to construct
an analytical result, researchers need to make more
definitive choices as well.
One strategy researchers can use to choose among
possible analysis paths is to reduce uncertainty.
They might follow a set of rules or procedures
in order to systematically select what they believe to be
a reasonable analysis.
For example, by using inclusion criteria
to scope their literature review.
This kind of thinking is necessary in order to
produce a result, but researchers really have to be
cautious, because not all instances of
the reduction strategy are equally justified.
The alternative strategy for making definitive choices
at a decision point is suppression.
Suppressing uncertainty is characterized by
ignoring possibilities,
gambling on potentially misleading analyses paths,
or rationalizing why potentially better alternatives
are not feasible.
Suppression is really the least desirable
decision-making strategy, but sometimes it is unavoidable,
and if the researcher needs to proceed with their analysis
under less than ideal circumstances.
In our interviews, we found that there's a blurred line
between reduction and suppression of uncertainty.
Although they are well-defined best practices
for research synthesis,
interviewees often describe shortcuts and compromises
to accommodate constraints,
such as limits on available time or evidence.
For example, one best practice is to collate evidence
across studies in a review by collaboratively filling in
a spreadsheet, called an evidence table.
Consider this quote from a different interviewee,
we'll call her Anna, describing whether she would
consider using an evidence table.
"As far as using it every time, and pulling out
every piece of information, I would literally have to
have an intern do that for me, because I don't have time."
We are interested in how software could make
systematic extraction of information into evidence tables
less cumbersome for researchers like Anna.
Now, you may be wondering if coding decisions
as suppression versus reduction was simply a value judgment
on our part.
We tried to avoid bringing our own beliefs
about appropriate procedures into play,
and instead we relied on participants' stated
or implied ideals, in order to decide whether or not
a decision should be considered suppression,
at least for the purpose of our analysis.
We found that the motivations behind researcher decisions
are often in conflict with one another.
For example, consider another quote from Joe.
"Initially, I wanted to include this topic in my review,
but there's only a couple studies to which that is relevant,
so it seemed like it was cleaner to just exclude."
When he says "cleaner," Joe seems to be concerned
about making inferences from a small sample.
Interestingly, a different researcher described
a similar situation where she made the opposite decision,
because the group of two studies that she was examining
were so vitally important to address her research question.
So through these two stories, we can see how the drive
to address research questions, and concerns about
statistical power, are in tension with one another,
in decisions about study inclusion.
We're interested in how interactive systems
could represent the trade-offs
between competing motivations.
So our analysis of interviews point to aspects of
the research process that should be represented
in interactive systems, and for the remainder of the talk
we're gonna discuss some ideas about what HCI researchers
can do to help with decision-making and research synthesis.
For starters, we've found it helpful
to formalize the problem, in order to ground our thinking
about how to design interactive systems
to support research synthesis.
We propose framing analytical decisions
as utility maximization.
Utility theory posits that the expected value
of a given action, for example, a gambling strategy,
is determined by the consequence of that action.
For example, the payoff of that gamble.
And the subjective value that we place on that consequence.
If we apply utility theory to researcher decisions,
we treat actions as possible analyses paths,
where each decision alternative entails an alternative path.
We treat the consequences of decisions as the accuracy
of analytical results produced by that path.
Note that in the formula above here,
we characterize the consequences of an action A,
in light of a state of the world S.
This is to say that the accuracy of a particular path
is determined in reference to the true state of the world.
But if we knew the true state of the world,
there would really be no need for analysis,
so this means that researchers must rely on proxies
to judge the consequences of their decisions,
because the true consequence of each path is unknown,
and perhaps unknowable.
These proxies might be their subjective sense of
confidence in their analysis, or how the results from
different analyses compare to one another.
Researchers also appraise the consequences of
their decisions through a value function,
which represents their rationale for decision-making,
and how they respond to these tensions
that we've been pointing out between goals and constraints.
Importantly, despite the fact that utility is
a formal model, the value function accounts for
the fact that rational and well-intentioned researchers
choose different analysis paths,
depending on their priorities.
Utility theory posits two fundamental requirements
for decision-making, which we can drawn on to think about
the design of interactive systems.
To ensure that they maximize utility,
researchers should be able to compare the consequences
of their actions, and they should be able to identify
the value that they place on those consequences.
So, we found three challenges
and corresponding opportunities for designing a support
for research synthesis.
In our interviews, we see that researchers are actually
inclined to survey multiple possible analysis paths,
in part because they're driven to
conduct a comprehensive analysis.
However, in practice they need to commit to a plan
and implement it, if they are going to achieve
a finished analysis in a reasonable amount of time.
Consider this quote from our Navy scientist, Anna.
"We'll have suggestions for these different types
of analysis we can conduct that might help us split the hair
a little bit differently, to give us additional information,
that we can then decide, okay, what is the best way
to report on this,
now that we've tried it both of these ways?"
The notion of splitting the hair a little bit differently,
to gather additional information,
might strike us as rigorous.
And at the same time, the notion of choosing what to report
after trying multiple analyses might strike us as
behavior akin to p-hacking.
This dilemma is illustrative of a particular challenge
for software design.
How should we support researchers
in trying multiple analyses?
One solution,
proposed by prior work and multiple statistics,
is multiverse analysis, which is essentially trying
and reporting multiple analyses.
And existing tools do not really support this kind of
analysis without the programming of custom analysis scripts,
which limits the accessibility of multiverse analysis
for researchers who don't code.
From the perspective of
enabling utility optimal decision-making,
comparing multiple possible results
through multiverse analysis could help researchers
calibrate their sense of the consequences
of their decisions.
So for example, when different analyses paths lead to
convergent results, the researcher may choose a path
that trades off accuracy with other desirable properties
like time, or resources saved, or maximizing consensus
or interpretability.
Alternatively, when results are highly variable
across analysis paths, researchers should exercise
extreme caution, and they should probably report
multiple analyses, if possible.
Another challenge for design is that despite best practices,
which emphasize consistent ways of processing evidence
from studies, researchers seem to document the rationales
of their decisions inconsistently.
If researchers document their reasoning at all,
it is often distributed across personal correspondences,
notes, and various work documents,
and there are multiple reasons for this,
among them the fact that researchers have limited time,
and the fact that researchers working within organizations
sometimes do not have the agency to reshape their objectives
in light of available evidence.
From the perspective of enabling
utility optimal decision-making,
inconsistently documenting rationales
may contribute to difficulty weighing motivations,
for the purpose of making these important value judgments
about the consequences of decisions.
One possible way to address this challenge might be
a system of user-defined and reusable flags
that researchers could use to attribute their rationales
for various analytical decisions, and these might be similar
to custom code books and Rayyan, shown here.
Rayyan is an existing tool for systematic review,
but there are some open questions here.
One, how to make this kind of feature flexible,
while at the same time ensuring that researchers
use it consistently.
Two, how to make such documentation minimally cumbersome
for researchers.
And three, how to represent trade-offs when
competing motivations suggest different courses of action.
A third major set of challenges are around how researchers
present uncertain information to stakeholders.
Consider another quote from an interviewee in the Navy,
describing what decision-making officials expect
from uncertain communications.
He says that decision makers do care about the fidelity
of the data, they just don't wanna chart on it.
However, by de-emphasizing this information
to satisfy a lack of tolerance for ambiguity and risk,
researchers may be suggesting that inferences have
greater credibility than they do.
We argue that tools should provide researchers with
communications techniques, which counteract decision makers'
specific aversions to uncertainty.
For example, what if stakeholders are averse to uncertainty
because they misunderstand probabilities
or statistical models?
Well, prior work in psychology, statistical reasoning,
and data visualization, suggests that people reason
about uncertainty most accurately when it is framed as
frequencies of events, rather than probabilities
or summary statistics.
For instance, visualization techniques
like quantile dotplots and hypothetical outcome plots
show possible outcomes as discrete graphical objects,
and this enables users to reason about uncertainty
by counting the proportion of dots,
for example in this image,
which represent desirable or undesirable outcomes.
In addition, frequency framings have also been adapted
in interfaces for listening users' predictions
about effect size and reliability,
so as to deepen their engagement
with uncertainty information.
Now alternatively, if stakeholders really just want
a clear yes or no answer,
and perhaps find these distributions overwhelming,
it may help the emphasize point estimates,
or alternatively, researchers could propagate uncertainty
and estimated effect distribution to derive measures
that maybe more closely align with stakeholder values,
such as cost or time saved.
So there's a large body of work on uncertainty communication
to the public, and for decision-making,
but this work is not well-integrated with the design
of interactive systems.
It is not absorbed to more recent findings,
such as those on uncertainty visualization,
and it has only been marginally codified
in researcher best practices.
So one of the goals of the recommendations of our paper
is to begin charting a research agenda for drawing together
these bodies of research in service of interactive systems
for data analysis and research review.
Some questions we plan to address in future work
on interfaces for meta-analysis concern
how researchers extract effect size information
from a heterogenous set of studies,
how they reason about the extent to which these effects
are comparable, how researchers conceive of the meaning
of the aggregated effect resulting from a meta-analysis,
and how to ensure that their intuitions align with
the actual statistical meaning of these estimates.
And we think that interactive systems have a key role
to play in supporting critical thinking and experimentation
around questions like these.
So to summarize, we found that researchers conducting
research synthesis need interactive systems which support
them trying multiple analyses, representing the reasoning
behind their decisions, and communicating uncertainty
and analytical results.
In the paper, we present a more extensive set
of opportunities, to address these design challenges,
as well as a thorough exposition of results.
And you can check it out by taking a picture
of this QR code.
Thank you for your time and attention.
(audience applause)
- [Moderator] Thank you, Alex.
So that was a long good talk, so we've only got time
for one quick question, if anyone has a question.
If there are no questions--
Oh, there is one question over here.
(unintelligible)
- [Pourna] Oh, hi, I'm Pourna from University of Notre Dame.
In the beginning, you said you asked the participants
two questions.
- You can go ahead and get set up.
- [Pourna] Do you ever do these things,
and then how do you go about doing them?
Do you have like the list of things you ask them?
- Right, so as I said, the interview guide
is a list of topics, and these were semi-structured
conversational interviews, so what we did was,
we would broach each topic according to sort of a
formulaic script, and then oftentimes there would be
a resulting discussion in which
we might ask follow-up questions,
to get at things that our interviewees
thought were particularly important to our study.
- [Pourna] How did you come up with the list of topics?
- Mostly based on prior work on researcher degrees
of freedom, what kinds of places in the analysis process
that researcher has the opportunity to make decisions
which impact subsequent results.
- [Pourna] And they were not domain-specific?
- Not particularly, no.