Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation
in Computational Notebooks.
I am April Wang, a Ph.D. candidate from the University of Michigan.
This is an internship project done together with my internship peers and mentors at IBM
research.
Computational notebooks allow data scientists to express their ideas through a combination
of code and documentation.
Unfortunately, data scientists often write messy and drafty analysis code in computational
notebooks as they need to quickly test hypotheses and experiment with alternatives.
It is a tedious process for data scientists to then manually document and refactor the
raw notebook into a more readable computational narrative, thus many people neglect to do
so.
Many efforts have sought to address the tension between exploration and explanation in computational
notebooks.
For example, researchers have explored the use of code gathering techniques to help data
scientists organize cluttered and inconsistent notebooks.
These efforts focus on the cleaning and organization of existing notebook content, instead of creating
the new content.
Another line of work looked into linking chat messages as documentation in the collaborative
data science space.
However, this approach is limited to real-time collaboration scenarios where data scientists
discussed frequently with each other as the notebook evolves over time.
We began our project by asking, “What does a well-documented computational narrative
look like?”
To answer this question, we first conducted an in-depth analysis of how data scientists
document notebooks.
Publicly shared user notebooks on Github are often not well documented, thus we look up
to a special set of notebooks—the highly-voted notebooks users submitted to Kaggle competitions.
We sampled 80 highly-voted Kaggle notebooks and conducted a qualitative content analysis
to explore their documentation practices.
We replicated the notebook level descriptive analysis by Rule et al..
As shown in the figure, the left side represents the descriptive visualization of the Kaggle
Corpus and the right side represents the Github Corpus.
We found that the Kaggle Corpus has more total cells than the Github Corpus.
The Kaggle Corpus has a roughly equal ratio of Markdown cells and code cells per notebook,
while Github Corpus is unbalanced with the majority of cells being code cells.
Notably, Kaggle Corpus has more total words in Markdown cells than Github Corpus.
This result indicates that the highly-voted Kaggle notebooks are better documented than
general Github notebooks.
Second, data science documentation covers a broad range of topics and purposes.
We coded nine categories of documentation in the Markdown cells.
For example, “process” describes what the adjacent code cell is doing;
“headline” is the styled headline for organizing the notebook into separate sections;
“result” describes the outputs from code execution; “reason” explains results or
critical decisions.
Lastly, we found that notebook documentation interplays with the stages of the data science
lifecycle.
We coded Markdown cells based on where they belong in the data science workflow.
Based on findings from the formative study and design insights from related works, we
design and implement Themisto, an automatic documentation generation system that supports
data scientists to write better-documented computational narratives.
Themisto is implemented as a Jupyter Notebook plugin.
Each time the user changes their focus on a code cell, the plugin is triggered.
When a user clicks on the button next to the code cell, Themisto will suggest documentation
generated by three approaches, which we will elaborate on later.
If the user likes one of these three candidates, they can simply click on it, and the documentation
candidate will be inserted above the code cell or below it.
Themisto generates documentation from three approaches.
Our formative study suggests that the system should be able to generate multiple types
of documentation.
Some types of documentation can be directly derived from the code, thus the automated
approaches can help.
For example, the Process type of documentation directly describes the coding process.
We trained a deep learning model using the Graph-Neural-Network architecture.
These GNN models can take both the source code’s structure, which is extracted as
abstract syntax tree, and the source code’s content as input.
Thus, it outperforms the traditional sequence-to-sequence model architectures, which only take the source
code’s content as an input sequence.
Next, our formative study indicates that the well-documented Kaggle notebooks often have
the description of frequently-used data science code functions for educational purposes.
Thus, we implement a query-based approach that curates a list of APIs and their descriptions
from commonly used data science packages.
Lastly, some others types of documentation such as descriptions of the results are not
directly related to the code.
We design the prompt-based approach for users to complete the generation process.
To evaluate the effectiveness of Themisto, we conducted a within-subject controlled experiment
with 24 data scientists.
The task is to add documentation to the given notebook.
And each participant is asked to finish two sessions, one with the Themisto support and
one without its support.
Our study found that Themisto can support data scientists in generating documentation.
For the sake of time, I would like to elaborate on the two key results from the study.
First, Themisto supports participants to easily add documentation to a notebook.
As shown in the table, we compared the task completion time and participants’ satisfaction
with the final notebook.
The results have shown that Themisto improved participants’ performance on the task by
reducing task completion time and improving satisfaction with the final notebooks.
The post-experiment survey result supported our findings.
Most participants believed it was easier to accomplish the task with Themisto’s help.
Looking into the post-task interview data, participants reported that Themisto provided
them something to begin with, thus it was easier than starting from scratch.
The second finding we would like to highlight is that co-creation yields longer documentation
and improves accuracy and readability.
Through coding the video recordings.
we found that while Themisto is available, for around 80% of code cells, participants
checked the recommended documentation by clicking on the light bulb icon to show the dropdown
menu.
Notably, around 40% of markdown cells were co-created by Themisto and human participants
together.
This finding inspires us to further explore how participants co-create the documentation
with Themisto.
By looking at the log data, we discover several editing patterns.
For example, many participants added supplemental details to Themisto’s suggested documentation.
Participants also added stylistic edits such as polishing sentences and changing conversational
tones.
Alright, if you are interested in more results, please read our paper.
In summary, inspired by human documentation practices learned from 80 highly-voted Kaggle
notebooks, we design and implement Themisto, an automated documentation generation system
to explore how human-centered AI systems can support human data scientists in the machine
learning code documentation scenario.