Hi, my name is Amy Zhang and I'm presenting
this work in collaboration with Michael Muller
and Dakuo Wang of IBM.
The focus of our work is on data science within organizations.
And how data science projects today
are often conducted in collaborative teams instead
of alone.
These teams often involve multiple roles
with unique skills and domain expertise not always technical,
and they contribute at different stages of the typical data
science workflow.
Despite this, many tools and studies
today still focus on individual data scientist.
Even when they do incorporate collaboration
they oftentimes focus only on technical roles
and technical stages of the workflow.
So in this work, we provide a richer account
of collaboration in data science teams
by examining a broader range of roles including
technical ones and non-technical ones,
and a broader set of stages of the data science workflow.
To gather data on collaboration, we conducted a study
in early 2019 with 183 people who
have worked in data science teams
within the organization IBM.
We asked questions about the roles people
collaborated with and in what stages of the workflow,
the tools people use for collaboration,
as well as people's practices and expectations around data
and code sharing and re-use.
So then turning towards results regarding data science
roles we overall found high self-reported rates
of collaboration across all the roles we examined.
People reported collaboration across roles
that were overall reciprocal and equally distributed,
with the one exception of communicators, who reported
high collaboration with others while other roles reported
low rates of collaboration with communicators.
Next, we asked about collaboration
each stage of the data science workflow.
We found that the role of researchers
are active at every stage of the workflow perhaps acting
to guide the project overall.
To a lesser extent, this was also true
for domain experts who may be consistently called on
to consult in every stage.
We found that roles like managers and communicators
are generally active in non-technical stages,
and not very active in technical stages,
and this was the opposite for engineers.
And in the case of bias mitigation stage in particular,
we are interested to find that this was mostly
treated as a technical matter with little involvement
from managers or communicators.
Turning towards tools we conducted qualitative coding
of people's free response questions
asking what tools they would use in different stages
of the workflow.
And came up with the following categories, anything
from Slack for discussion, to Python for coding,
to Box for file sharing.
Then we map these categories across stages of the workflow.
Some things were unsurprising, like bad code
related tools seem to fill in for asynchronous discussion
during the more technical stages of the workflow.
We also noted relatively lower usage
of tools related to documentation, presentation,
or discussion in some important stages.
Where a lot of consequential decisions get
made such as future engineering and bias mitigation.
To better understand how the use of certain tools
relate to collaborative practices,
we clustered our respondents according to what tools
they stated they used.
We came up with three main tools profiles
including project managed, and these
were people who heavily use project and team oriented
tools like GitHub and Slack.
Interactive, who mostly used interactive code editors
like Jupyter Notebook.
And also the scripted profile, who
made greater use of one of scripts in Python this cluster
was also mostly engineers.
We found that across all profiles,
people stated they read and re-used other people's
code more often than other people's data.
But interestingly, when we asked about their expectations
that other people would use their code or data,
the project manage profile actually had high expectations
around data re-use while the interactive and scripted
profiles had low expectations around data re-use.
This interestingly carried over to documentation practices
which was overall lower for data than code though we did
see the project manage profile made
use of the strategy of external documents
pretty frequently to document data.
So in summary, we found that data science workers
collaborate extensively and in every stage of the data science
workflow.
Documentation and discussion, especially
with non-technical roles may be insufficient in certain stages
like feature engineering and bias mitigation.
Practices, expectations, and toolings for sharing and re-use
around data is less developed than for code.
And in conclusion, we need more data science tools
that can capture the provenance of and decisions
and discussions around both data and code
for both technical and non-technical collaborators.
Thank you and you can contact us here.