My name is Steven Kasica, and I'm presenting Dirty Data in the Newsroom:
Comparing Data Preparation in Journalism and Data Science.
This is working in collaboration
with my coauthors, Charles Barret at the University of Linköping,
and Tamara Munzner at the University of British Columbia.
Preparing data is often the most time consuming and tedious aspect of data work,
including wrangling, cleaning, munging, and gathering data.
Many researchers have studied this process among data scientists.
Finding that this can take up to 80% of a project's overall time.
Today, journalists are increasingly turning to data as a source
to deliver the day's news in a new way and to support investigations
in ways that traditional newsgathering techniques cannot.
While the field of journalism has historically gone through
many quantitative turns, the work today falls under the genre
of data journalism.
Compared to the data sciences,
not much is known about how these data journalists prepare data.
But we do know that they use
many of the same tools from the data science community.
So our study seeks to understand how closely the abundant research
and data preparation in data science applies to data
journalism.
We conduct our study in four phases.
In the first phase, we analyzed 16 research papers documenting
data scientists preparation workflows which were identified in related work,
conducting a literature review in this area.
And we use thematic analysis to analyze this data
based on the activities these data science workers performed
and the underlying data issues these activities address.
In the second phase, we interview 36 data journalists
on their preparation practices, and we analyze these interview transcripts
with regards to activities and issues,
noting where data science codes from Phase 1 apply to this data,
but also generating new data journalism codes from this interview
data. From our analysis of these activities, we propose
an augmented model of preparation activities--our first contribution. But
we felt like our analysis of the issues was incomplete.
So we consult a third source.:
16 taxonomies and dirty data published in the research literature and databases.
We reflect and synthesize
on these issues from data science, data journalism and databases
to identify 60 distinct data issues
and categorize these into a new taxonomy--our
second contribution. In the fourth
and final phase review are interview transcripts at a higher level.
All participants told us in detail about particularly challenging data preparation
stories, and we note 63 nightmare data stories participants described.
We categorized these stories
into four challenges involving multi-table data integration.
Our third and final contribution.
Our first contribution is an augmented model of preparation
activities. In the first two phases,
we analyzed papers and data science workflows in the transcripts
from our interviews with data journalists with regards to preparation activities.
And from this phonetic analysis, we identified 23 preparation activities.
We also match our findings to a model of data science work
proposed in related work by Crisan et al.
And this model
nicely frames data preparation in both data science and data journalism.
And hence we call this an augmented model of data preparation activities.
And we're able to contrast and compare both groups
by looking at the occurrences of codes in these two qualitative data sources.
We note
activities that were mentioned in the data science process, papers
that were never or rarely mentioned by data journalists.
And likewise, we knew that activities that were often reported by journalists
that are underreported in the data science literature.
Our second contribution is a model-discrepancy taxonomy of dirty data.
We begin by considering data as a design artifact and dirty data
as discrepancies in mental models between data workers.
We extend our issue analysis from the first two phases to incorporate
issues from the database literature analyzing 16 Taxonomies on dirty data
clustering over 300 issues into 45 unique issues in this corpus.
And then we combine this with our issue analysis of data scientists
and data journalists to create a synthesis set of 60 data issues.
Then we categorize these into a new model discrepancy taxonomy.
And these categories exist on two axes, the first being data qualities.
We extend existing qualities, completeness and accuracy with new qualities,
form granularity, relation and semantics.
And our second axis is data objects,
which includes tables, attributes, items, and values.
And more details are provided in the paper.
Our third
contribution are challenges in multi table data integration.
We classify the 69 data preparation nightmare stories
journalists told us and the four categories excluding six,
which do not fit into one of these categories.
And these categories are important with regards to data work in general
because they flip the script
on integrations role within the overall preparation process
instead of being viewed as a means to an end, often early in the process.
These stories treat integration as the end goal itself.
Our first challenge is regional datasets,
which come from independent sources, publishing data
on the same phenomena.
And one area where this turns up
is in reporting of use of force incidences by police officers.
When journalists told us of analysis involving the 50 largest
police departments in the United States, which also involved integrating
50 often very different datasets from these individual police
departments.
Our second category is diachronic datasets, tables on the same phenomena.
But the can change in subtle and significant ways over time.
Multiple participants discuss
analyzing employment data from the Bureau of Labor Statistics.
And one aspect of this challenge has been the values definition changes over time.
For example, a computer
analyst in the 1980s isn't the same thing as a computer analyst today.
And this can complicate overtime analysis.
Fragmented data as
are tables with different yet related items.
Coincidentally, multiple interviews
touched on investigations of unpaid safety violations in coal mines,
but linking safety violations to mine owners is not a straightforward task.
Safety violations are often linked to data on mine locations.
Mine locations are operated by mine operators
who are employed by mine owners.
And these data sets need to be assembled carefully
to support accurate and valid analysis.
But this work is important because mine owners sometimes
are prominent politicians.
Disparate data sets are typically dissimilar and seemingly unrelated.
One powerful example of this challenge involves
one participant covering the opioid epidemic
by taking data that contained individual state health
care workers with a separate data set on statewide deaths.
This journalist was able to match entities by name
in order to identify health care workers who died of opioid overdoses.
In this data, work provided leads that the journalist
followed up with using traditional news gathering techniques.
We compare how journalists and data scientists prepare raw data.
We contribute an augmented model of preparation activities,
finding that data scientists and data journalists have a lot in common.
We propose
a new model discrepancy taxonomy for dirty data, which frames existing
issues in a novel way that better matches how end users think about their data.
And finally, we identify four challenges in multiple integration,
which itself challenges the model
for how data integration fits within the overall preparation process.
Thank you.