Good afternoon everyone.
I’m Grace Guo from Georgia Tech, and today I will be presenting our paper Causalvis:
Visualizations for causal inference.
When we think about causality, we are usually referring to how a certain treatment affects
a particular outcome.
Typically, when we hypothesize that two variables share a causal relationship, we would test
our hypothesis by running an experiment, otherwise called a randomized controlled trial.
However, not all hypotheses can be tested in this way.
What happens when RCTs are infeasible?
Consider this hypothesis.
Does smoking cause lung cancer?
For many good reasons, we cannot ask participants to smoke cigarettes and test if they develop
lung cancer.
In such situations where RCTs are unethical, costly or infeasible, we must rely on observational
data.
Which brings us to causal inference.
Causal inference is a statistical paradigm for quantifying causal effects using observational
data.
Consider again our hypothesis that smoking causes lung cancer.
If we have access to observational data sets collected by hospitals or healthcare organizations,
we can look at lung cancer rates in smokers and non-smokers.
But simple comparison is not enough to establish a causal relationship.
When working with observational data, additional steps must be taken to ensure that the estimated
treatment effect is reliable and unbiased.
In this example, age might be a confounding factor that affects both smoking and lung
cancer.
If smokers tend to be older than non-smokers, we can no longer conclude whether the change
in cancer rates is caused by age or by smoking behavior.
Identifying and adjusting for biasing variables, such as confounders, is crucial to causal
inference.
Only after adjustment, can we then compare outcomes between the two groups to obtain
the average treatment effect.
The ATE quantifies the size of the effect of the treatment on the outcome variable,
and is often the main goal of causal inference.
But as we have just demonstrated, causal inference is a complex process with multiple steps and
challenges.
So in order to better understand user tasks and to develop visualizations that support
their work, we conducted a design study with 8 causal inference experts and researchers.
After two rounds of interviews, we distilled the causal inference workflow into three main
steps.
Starting with observational data, the analysis begins with causal structure modeling, followed
by cohort construction and refinement and finally, treatment effect exploration.
We also identified 8 user tasks, which informed the design of the 4 Causalvis visualization
modules.
I will next introduce the first 3 modules.
For the last, VersionHistory module, I refer everyone to the paper for more details.
To better demonstrate Causalvis, I will be presenting in context of a usage scenario.
In this scenario, we look at the UCI Student Performance data set which records student
math grades and related data at two Portuguese schools.
We are interested in estimating the effect of absences on grades.
Here, absences is the treatment variable and grades, or G1, is the outcome variable.
We begin with the first step of causal structure modeling.
Causal structure modeling is the process of identifying causal relationships between variables,
typically represented using directed acyclic graphs (DAGs).
The goal of this step is to identify the right covariates, such as confounders, for statistical
adjustment.
Causal inference analysts would typically collaborate with domain experts at this step.
For example, a teacher or an educator would be most qualified to identify factors that
influence student absences or exam grades.
But teachers and domain experts in general are often unfamiliar with causal inference
and related terminology.
So the causal inference experts in our study mentioned that they would like a tool to facilitate
collaboration with domain experts.
This led us to develop the DAG visualization module.
In our scenario, we can pass the student data set to the DAG module to manually create a
DAG from scratch.
Users can collaboratively add variables to the graph, position them and edit the links
between them.
Once we have a DAG of causal relationships in our data, we can set the treatment and
outcome variables.
Here, we set absences as the treatment variable, and G1 as the outcome variable.
The DAG automatically updates to highlight the other variable types.
From this, we can identify that there are six confounding variables, and four prognostic
variables that should be adjusted for.
Together, these variables are called covariates.
The next step of causal inference is cohort construction and refinement.
In this step, we want to select our treatment and control samples from the observational
dataset.
In order for a cohort to yield a valid causal effect estimation, the distribution of the
covariates between the treatment and control groups should be similar.
There are different ways we can adjust for covariates.
In this example, we will use the matching method.
At a high level, the intuition behind matching is that given all samples in the dataset,
for every individual in the treatment group, we match them to an individual from the control
group with similar covariates.
However, some samples may not have a well matched counterpart.
In which case, we want to refine the matching algorithm to discard these samples.
But how can we evaluate for covariate balance?
That is to say, how can we evaluate the matching outcome?
From the previous DAG creation step we know what covariates need to be matched, but it
would be difficult to evaluate each match manually by comparison.
So one of the metrics we can use is the propensity score, which is the probability of each sample
to be exposed to the treatment variable given its covariates.
Since the propensity score is a summary of the covariate space, we can compare propensity
score distributions between the treatment and control groups, and thus evaluate whether
the groups are sufficiently similar.
If we want to look at specific covariates, we can also compute the absolute standardized
mean difference score, which quantifies the difference between the treatment and control
group for a single covariate.
Returning to our scenario, we used matching to select students from the UCI student performance
data set, we obtain a cohort with 200 samples.
We then pass this cohort to the CohortEvaluator module of the Causalvis package.
The module visualizes propensity score distributions on the left, and aSMD scores on the right.
Here, we see that the distributions of treatment and control group propensity scores are similar.
We also see from the aSMD plot that the adjusted aSMD values for all covariates are below 0.1.
This threshold is typically used to identify when variables have similar distributions.
This is a well-balanced cohort, which we can now use in subsequent treatment effect estimation.
The estimated ATE we obtain is 0.42.
Students with frequent absences had slightly higher grades than students with few or no
absences.
But this is a very small difference, and does not mean that either group is doing better
than the other.
Which is also why causal inference analysts often want to know more than the ATE for the
entire cohort.
This brings us to the third and final step of Treatment Effect Exploration, where analysts
explore and compare subgroups in the data set.
Differences between subgroups indicate heterogeneous treatment effects, and even instances of the
Simpson’s paradox.
For our selected cohort, we calculate individual treatment effects.
This is passed to the TreatmentEffectExplorer module.
The distribution of individual treatment effects is visualized using a raincloud plot.
We see that the distribution is centered around zero, which matches the previously estimated
ATE of 0.42.
However, we can explore this data set further.
We can choose to group the cohort based on internet access and student health.
Across the four subgroups, we can see immediately that students with poor health who had no
internet access at home had a clear decrease in grades caused by frequent absences.
Comparatively, this effect was less pronounced for the other subgroups.
This is thus a heterogeneous treatment effect, and may be an interesting finding for stakeholders
in our scenario, such as parents or educators.
Finally, causal inference analysts often repeat their analysis with different DAGs and cohorts.
We developed the VersionHistory module to save the iterations, but I won’t demonstrate
the module here.
I refer you to the paper and demo video for more details.
We have now seen the Causalvis modules and how they correspond to the causal inference
workflow.
We then went on to evaluate the visualizations with 11 participants.
Overall, participants felt that we addressed the eight user tasks identified earlier.
The detailed feedback are reported in our paper and I won’t reiterate them here.
But I want to discuss a couple of interesting findings from the evaluation.
Firstly, the iterative nature of causal inference means that analysts typically need to test
multiple hypotheses.
Our users appreciated how the interactive and modular implementation of the visualizations
meant that they could repeat steps and refine their hypotheses really rapidly.
A second interesting finding was that many participants valued how the visualizations
helped them explain and communicate causal inference to collaborators.
They also suggested adding annotations to further support this goal.
In particular, some participants emphasized the importance of being able to tell a story
about causality.
In our paper, we discuss these and other findings in greater detail.
Finally, we also learned some lessons from our design study about
The importance of visualizations that complement instead of replace existing workflows,
How to prototype and design for computational environments such as Jupyter lab,
As well as the importance of working with diverse collaborators in order to understand
difference usage scenarios and tasks.
This brings me to the end of my presentation.
I would like to thank my co-authors Ehud, Alex, and BC, as well as IBM research and
GT Visualization Lab for their feedback and support.