I'm Eunice Jun, and I'm excited to talk to you about our work in defining hypothesis
formalization. The curiosity that inspired this work was this very simple question:
How do researchers author statistical analyses? Specifically, how do they go from research
questions and hypotheses to specific statistical model implementations in code? To begin answering
this question we turned to prior work. Prior work in HCI has conducted interviews studies
with data workers, where they've uncovered that data analysis is an iterative process with various
different types of analysis alternatives and paths that may be explored. In prior research in
statistics has argued for statistical analysis as a sense making activity where researchers
and analysts are involved in testing and updating their mental models of the world based on insights
derived from data. Synthesizing this prior work in HCI and in statistics, we develop this initial
model of data analysis praxis, which involves conceptual hypotheses, implicit causal models,
the importance of data and observations, and their influence on statistical specifications.
In addition, prior research on theories of scientific discovery has posited that
scientific discovery exists in four spaces. So this is Schunn and Klahr's 1995 model, where
there are four spaces or considerations involved in scientific discovery: hypothesis, experiment,
paradigm (which includes model implementation), and data representation. Importantly,
there's only a unidirectional arrow from hypothesis to paradigm (or model implementations),
even though prior work suggests that this diagonal may in fact, be bi- directional. So in our
research, we connect these two bodies of research on data analysis praxis and theories of scientific
discovery. And we extend existing theories to show how hypothesis formalization exists on this
diagonal from hypothesis to paradigm, and, in fact, includes a bi-directional relationship.
We divided our initial motivating question into three specific research questions.
First, what is the range of steps an analyst might consider when formalizing their hypotheses?
Second, how do analysts think about and perform the steps and finally, how might current software
tools influence hypothesis formalization. Content analysis findings reiterate and
emphasize some of the key phases identified in prior research is just conceptual hypotheses,
implicit causal models and the importance of data. But in addition, our content analysis finds that
researchers are very focused on identifying proxy variables for how they measure their concepts.
And in fact, they break down their conceptual hypotheses into specific sub-hypotheses.
And they iterate on this process in a hypothesis refinement loop. So we have initial evidence for
this iterative process involving hypotheses, variables, and sub-hypotheses. In addition,
just as prior work has suggested, there's an important relationship between datasets and
the statistical model implementations, starting to give us a clue into the dual search nature of
hypothesis formalization that not only involves hypotheses and data but also statistical model
implementation. We conducted a lab study to answer our first two research questions and
recruited 24 data workers. The study was broken down into three parts. The motivating question
posited to the participants was "What aspects of an individual's background and demographics are
associated with income after they have graduated from high school?" The three part study we first
asked participants to brainstorm hypotheses, conceptual models, statistical specifications in
the lab, and then implement these statistical analyses at home, and then come back to the
lab in order to reflect on their analysis process, the results and their experiences.
Some key findings from our lab study include a reiteration, from our content analysis, that
researchers focus on proxies and data collection when articulating hypotheses---thinking about
surveys or other available datasets, for example. We were also surprised to see just how much
participants were fixated on implementation and statistical tools when asked us to specify
and discuss their statistical approaches. So for example, one participant outlined a
very detailed list of steps that they would take in order to conduct their statistical
tests, which is, you know, only mentioned at the very end. They include everything from data
transformation to wrangling to visualization. And addition, we were really surprised to just see how
focused researchers and data analysts were in fitting their current statistical analysis to
previous projects and familiar approaches. One participant described how they usually
tend to jump into their data and match the analysis problem with similar patterns they've
seen in the past and start implementing that after doing some rough diagramming.
One researcher went so far as to actually just generate data for the task. So for instance,
they described how they really don't like non-normal data, and then their normal practice,
they really try to do everything they can, whether it's, you know, transforming data, cleaning
outliers, etc, to get normal data, just because non-normal data messes their process up so much.
Although, you know, generating synthetic data may be acceptable for a lab study
and not something that this researcher does in their normal practice,
it nevertheless illustrates just how adherent analysts are to ways that they are familiar with,
even if they're sub-optimal, and in fact undermine their current analysis goals.
Furthermore, we found that analysts were really focused on, you know, avoiding or
resisting stating their assumptions for fear of biasing their analysis. Right, as an illustration,
one participant, you know, described how income would plateau after a certain age,
yet well, another participant described how income would continually monotonically increase with age.
Both participants thought this, you know, relationship between age and income
was so obvious that they didn't need to, you know, state it, and that, in fact, by stating it,
somehow they would bias their exploration of their data. But nevertheless, right you have two
contradicting views on what income what happens to income over the course of one's lifespan. When
probed a little further, one participant described how it's so easy to run individual tests,
you can run hypothesis tests faster than you could actually think of a hypothesis might be.
So there's no need to really presuppose what relationships might exist. Interestingly,
right from the same observation, we might think that because statistical tests are so easy to run,
it's more important that we state our assumptions and our hypotheses a priori.
This goes to further highlight the importance of both of these searches in hypothesis
formalization, which we described to be a dual search process that involves hypothesis
refinement on one end and statistical model implementation iteration on the other. And
that this process is really a problem solving and design process where analysts must reconcile
constraints and considerations that arise from their concepts, their data,
statistical methods, and actual implementations. So, we began to kind of probe further into what
does that actually mean for how tools are either supporting or taking away from this process.
We focused on four key aspects of tools: their specialization and scope,
their model expression, the statistical taxonomies they embodied as well as the computational control
they offered end users. And here, I'll just highlight a few key illustrations of our findings.
So the first is that specialized tools are really important for creating this diverse ecosystem of
statistical methods and tools. But they also come at the cost of pushing computational concerns
higher up the hypothesis formalization process. And so statistical tools also introduced syntactic
and semantic mismatches that could create a rift between model implementations and conceptual
hypotheses. An illustration of this was in the ANOVA, is what the sums of squares procedures are,
basically how is it variance is partitioned among variables specified. So in the R stats core
package, the default is to use type one. And in the statsmodels package, its default is type two,
whereas in SPSS, it's, you know, type three. And these three different sums of squares, procedures
lead to different F statistics and p values, which may lead analysts to different conclusions.
But more importantly than that, they actually test very different conceptual hypotheses.
Furthermore, we found that tools require analysts to match their conceptual hypotheses with tools'
taxonomies, which can be idiosyncratic. In fact, we often found misalignment and inconsistencies
between these taxonomies. These low level control knobs that these tools provide end-users, such as
specifying a number of threads or specifying the number of CPUs to run on are helpful, but
they also lack feedback. So for example, analysts can easily change parameters to fine tune their
settings. But then this might actually require conceptual revisions, that introduces a gulf
of evaluation. So this goes to further emphasize just how important this dual search processes is,
right how the current tooling ecosystem is really focused on this lower portion of hypothesis
formalization even though end users really care about their concepts and all the things that
lead up to that point. So how might we go about bridging these gulfs? Based on our observations,
we derive two design implications for tools. The first is around high level abstractions.
Tools should provide higher level abstractions for expressing model implementation intents.
Higher levels of abstraction themselves are not enough, but they must also be coupled coupled with
computational assistance that, for example, might include more generic interfaces for libraries with
all slightly different modeling syntaxes so we connect model implementations with mathematical
equations, and show and underscore some of the statistical similarity between seemingly different
methods, such as the ANOVA and linear mixed effects model. Or, another opportunity for
high level abstractions is to allow end users to more directly express their conceptual hypotheses
and then to bootstrap the model implementation process. A second implication of our work is to
really focus on co- authoring conceptual and statistical models. Rather than focusing end
users on statistical models and expecting them to do all the work are priori to get to a specific
statistical specifications. How can we urge our tools to consider this entire process end to end?