Diff in the loop: Supporting data comparison in exploratory data analysis.
I am April Wang, a phd candidate from the University of Michigan.
This is an internship project done together with Will Epperson, Robert DeLine, and Steven
Drucker.
Data science programming is characterized by evolution.
In particular, data scientists spend up to 80% of their time doing exploratory data analysis
tasks like cleaning, filtering, and formatting data tables before they can begin further
analysis.
Since this process is highly exploratory, results evolve from moment to moment.
In the meanwhile, this process can be collaborative, meaning that results can evolve when the work
changes hands.
Tracking and making sense of the changes are critical for exploratory data analysis.
For example, many development environments allow users to compare differences in data
science code between different revisions.
Recent studies have demonstrated the benefits of interaction with analysis histories during
exploratory programming.
Kery and her colleagues designed an algorithmic and visualization approach for finding past
exploration paths in the long editing histories of computational notebooks;
Head used program slicing to gather relevant editing histories into a minimal notebook.
Existing tools for tracking changes typically only tackle half of the problem: differences
in code.
Yet comparing versions of data throughout analysis is just as important.
Code differences do not always reveal data differences.
For example, just by looking at the two versions of code, it is not intuitive how much
the log transformation has changed the shape of the data.
To track the effect of the code on the data, data scientists have to take the initiative
to write additional code to browse or plot the data.
What if visualizing differences in data iterations becomes a first-class citizen in data science
programming environments?
In this paper, we explore adding visualizations of data differences as a core feature of tools
for exploratory data analysis, a concept we call Diff in the Loop (DITL).
To motivate the problem, let’s look at three typical usage scenarios that demonstrate how
showing both code and data differences would be useful during exploratory data analysis.
First, debugging exploratory analysis code involves both ensuring that code changes can
compile, but also produce expected results.
Data science programmers often need to write temporary code queries to inspect data tables.
They may miss unexpected impacts if they do not thoroughly explore the effects of code
changes.
These code queries are likely to become stale or commented code which reduces the readability
of the analysis notebooks.
Therefore, we believe that showing both code and data differences as first-class citizens
in data science programming environments can directly aid debugging.
Second, data scientists must make decisions throughout exploratory data analysis.
Which features should be taken into consideration?
How should null values be filled in?
Does it matter if this part of the data is dropped?
These decisions require making comparisons between alternatives.
We believe that comparison is an inherent task within exploratory data analysis.
Lastly, comparing data tables improves data scientists' awareness of each others' work
in collaborative settings.
Code versioning mechanisms in traditional software engineering can help data scientists
manage code iterations when handing off work.
However, it is not straightforward for data scientists to interpret the impact of code
changes, unless they execute the code and inspect the data tables thoroughly.
We believe that showing and tracking both code and data changes can augment the existing
data science collaboration tools by improving awareness of changes.
To address these use cases, we present DITL, a process of inspecting and comparing versions
of data tables using interactive visualizations.
We integrate the design into a simplified data science programming tool
which allows data scientists to edit code, track versions of edits, inspect data table values from historic edits,
and compare different data tables.
Let’s look at an example for comparing data tables.
After loading the car dataset, we decided to filter data points with cylinders more
than 4.
DITL allows us to inspect the changes we made immediately through interactive visualization.
Users can switch between three approaches to display the diffs.
The delta view contains two parts -- users can switch between the current and original
distribution in top part, while seeing the subtraction between the current and original
distribution in the bottom.
The opacity view overlays the current and original distributions on the same axes and
provides a slider for crossfading between current and the original distributions.
Lastly, the parallel view shows the original and current distributions of the column side
by side.
It simulates the go-to strategy that data scientists manually use to plot and compare
the current and original distributions.
In addition, DITL highlights the changes in summary statistics for comparing the data
tables.
By looking at the Cylinder column, we can verify that the filtering code is working
properly as we expected.
DITL allows data scientists to do more than verify execution results.
It can also provide additional insights into the dataset.
For example, let’s look at the column Horsepower.
The visualization tells us that the distribution of Horsepower shifts to the right after filtering
cylinders.
Such side effect changes are often hard to expect just by seeing the code changes.
We conducted a 60-minute long virtual usability study with 16 professional data scientists
to understand the support that DITL can provide for common data science programming tasks.
We gave participants two existing data science tasks modified from online data science challenges.
We counterbalanced the order of the tasks between subjects.
For each task, participants are randomly assigned to solve it with or without the DITL.
Our evaluation showed that comparing data tables is a common activity in various data
science tasks.
During the study tasks, we frequently observed participants comparing data tables for various
purposes.
In addition, most participants agreed that they often need to compare two data tables
in their own work.
Most participants agreed that DITL makes comparison easier.
We synthesized the reasons mentioned in the post-task reflection.
One advantage of DITL is that besides reducing the amount of code that participants wrote,
it also eliminated code that was there just for verification or validation purposes.
Participants also mentioned that DITL helps them discover insights about the data, describes
the tool as “directly explains what is going on", "allows me to instantly look at the differences",
"is helpful for formulating the next steps".
In addition, participants mentioned that the visualizations helped them understand the
side effect of code edits.
Last by not least, participants from the usability study mentioned that they would like the idea
of DITL integrated into their work environments.
Next, we would like to reflect on the integration of DITL in common data science programming
environments.
To integrate DITL in existing data science programming environments, both scalability
and task complexity must be considered.
In particular, the timescale for creating snapshots of the data iterations should be
tailored to the context.
For example.
Programming IDEs can leverage built-in debuggers to track variables upon each execution.
For programming environments that allow interactive execution of code snippets, mapping the versions
of variable values with the state of the notebooks can be a challenging task.
Future work can explore how approaches used in foraging code versions can be extended
for foraging data iterations.
Lastly, the idea of comparing data changes can be helpful in live programming environments.
Live programming updates the execution immediately upon editing.
Although live programming is favored for providing a responsive and consistent experience for
exploratory data analysis, the live experience hides history and may result in mismatched
expectations for the automatic execution.
Future work can explore the idea of showing both code and data iterations in live programming
environments for browsing histories.
In summary, This paper presents the idea of Diff In The Loop (DITL), integrating data
differences through visualizations as a first-class citizen in data science programming environments.
To learn more, please read our paper.
Thank you :>