- Next up, we've got Jeremy Singer-Vine.
You may know him from his data investigation work
as data editor through Buzzfeed,
or like me from five years running Data Is Plural,
a weekly newsletter with interesting datasets running
from open government statistics,
all the way through to the color distribution
of Skittles and a database of canine cosmonauts.
Jeremy is gonna talk to us about data journalism
as a specialty and in particular FinCEN,
which is being used to investigate
international money laundering and financial crime.
Welcome, Jeremy.
- Hello, I'm Jeremy Singer-Vine.
And for this talk the most relevant fact about me is
that I'm the data editor at Buzzfeed News,
where I run a small team that collects and analyzes data
for the purposes of journalism.
Today, I'll be talking
about the largest reporting project I've ever worked on.
the FinCEN files, which we published this September
in collaboration with the
International Consortium of Investigative Journalists
and more than a 100 news organizations around the world.
You can use this QR code or link to read our full coverage.
Today I'll use the FinCEN files as a lens
through which to talk somewhat more broadly
about data journalism and how the work we do might differ
from the work in your field of expertise.
Work on the FinCEN files began more than a year ago.
Buzzfeed News received a remarkable collection
of secret government documents related
to suspicious financial transactions.
This huge trove had been assembled
at the request of law enforcement agencies
and congressional committees,
investigating the 2016 presidential election
and other matters.
The documents were so remarkable and globe spanning
that we began to talk with ICIJ,
the International Consortium of Investigative Journalists,
the organization that published the Panama Papers.
We talked to them about working
to "scale" the reporting if you will.
By the end of the project, more than 400 journalists
in 88 countries have pitched in.
We found that in these documents alone,
big banks had approved trillions of dollars of transactions
that they later told the government seems suspicious.
These banks failed sometimes
to take even the most basic steps
to root out financial crime,
such as determining whose money they were actually moving.
We published specific findings about the conduct
of HSBC, Deutsche Bank, Standard Chartered
and other major financial institutions.
We also unearthed new details about major scandals,
such as the so-called mirror trends.
But now I've jumped directly from the beginning to the end,
skipping 99% of the effort.
So much work went into this project
and I don't have time to talk about it all,
not even a tiny fraction of it,
but I do have time to talk a little bit about the data.
Before I do though, I think it might be helpful
to talk a little bit more about how I think
about data in a newsroom like Buzzfeed News.
Many of you are likely familiar
with the term data journalism,
it's become quite popular.
When many people think of data journalism,
they think about data journalism as a product.
Articles full of numbers and charts,
and articles that focus on numbers.
At Buzzfeed News we do some of that for sure,
but much more of our effort goes
into something I'd call data for journalism.
Data journalism as a process.
And that was definitely true of the FinCEN files.
Although we did publish an article
with some numerical findings,
our main goal was not to produce a data journalism product,
but instead to use our skills
to help our colleagues understand the documents,
to use data and data analysis to find interesting stories
within the documents and to put those stories in context.
Another reason I'm sometimes skeptical
of the phrase data journalism is
that the data rarely stands alone.
Instead it's supported by and supports a huge amount
of other journalistic work conducting interviews
for example, reading lawsuits, filing public requests,
public records requests and so on.
So our goal as journalist with data skills working
on the FinCEN files was really to ask ourselves,
"How can we make this information most useful,
not just for numbers, but for the project as a whole?"
What materials did we have to work with?
Well, there was very little machine readable
structured data, and this was pretty common.
We rarely start out with some amazing, perfect dataset.
Instead, we often have to work to get the data we need.
You may find this situation familiar.
There were some CSVs in the documents
but they were sort of at the periphery.
Instead at the heart of the caucus were more
than 2,100 suspicious activity reports,
which are official notifications
that U.S. banks must send to the treasury department
when they observe transactions that seem suspicious.
For example, the transactions might carry hallmarks
of potential money laundering or other financial crime.
Banks file millions of these per year.
We only had 2,100, but that was still magnitudes more
than had ever comes to public light before.
And these documents were incredibly detailed.
Here's an example, so it have two main parts.
The first is a set of tables at the top of the document.
This is an example here of just the beginning
of one of those tables.
And that's just one SAR.
Imagine this one going on for several pages,
sometimes dozens of pages.
The tables lists the people and organizations involved,
the overall dollar amount of suspicious activity,
the general timeframe, the bank that filed the report,
and more.
Here's an example of one of the narratives.
Again, just the beginning of one.
Some of the narratives are very short
but others include a lot of helpful details.
And unlike the tables, narratives often list
individual transactions,
rather than just the total aggregate amount.
And they often specify the banks and senders
and recipients involved in those individual transactions.
So where did we go from there?
The most obvious thing we needed to do
from the very beginning was
to liberate the data trapped in those tables
at the top of each PDF.
This work was in many ways familiar territory,
data liberation, whether it be web scraping, PDF parsing
or other techniques is sort of our bread and butter.
We do it very often.
We have at Buzzfeed News parsed PDFs to create new datasets
for investigations into foster care providers,
figure skating judges and other topics,
because the SARs were quite standard tables.
They involve several layers of nesting
and they contain some idiosyncrasies.
We wrote a custom parser using pdfplumber,
an open source library I've been developing
for tasks like this.
We then converted the raw representation of that data
in those PDFs into a series of so-called tidy data tables.
So we could quickly run new queries
in a broad variety of environments.
These tables were just plain CSVs.
We're talking about megabytes of data,
not gigabytes or terabytes.
The data from the tables helped us answer questions such as,
which banks appeared most often?
What people and companies were listed
as subjects more than once?
As well as more intricate analysis,
tailored to specific stories
or specific reporters interests.
I'd like to give a special shout out
to a couple of things in our step.
Now, a lot of the tools we use on the FinCEN files
and more generally are pretty well-known,
Python, Jupiter, the pandas data analysis library,
Git for example.
But there are a couple of things perhaps less known
that I'd like to highlight and to recommend.
One is just a concept.
A few moments ago I mentioned tidy data.
It's a particular way of structuring datasets.
Some of you may already be familiar with this concept,
popularized by Hadley Wickham of Tidyverse fame.
But if you're not, I highly recommend reading
his excellent 2014 paper on the topic.
Here's the link to that paper, tidy data,
both the concepts and the practice
have deeply changed the way I think about
and how I work with data on a daily basis
on every project I work on.
The other thing I'd like to highlight is a piece of software
and that's VisiData.
It's a tool for exploring
and analyzing datasets on the command line.
I've been using VisiData for years now
and have even begun making small contributions
to the code base.
It's gradually becoming more popular
but I think there's still a lot of people
who still have never heard of it,
but would really enjoy using it.
And again, here's a link to VisiData's homepage.
I use VisiData practically every day I was working
on the FinCEN files.
And because it's extensible I could build custom commands
specific to the project.
All of our final analysis ended up scripted
in Python, fully reproducible,
but VisiData was an invaluable tool
for quickly searching through, sorting,
and joining the data in the FinCEN files
and for quickly testing out new hypothesis.
So back to the SARs.
If the tables were in some ways relatively easy
to work with, what about the narratives?
Those freeform sections of each SAR
that provide additional details.
Those posed a greater challenge.
As I mentioned earlier, we were most interested
in capturing the individual transactions
that the narratives discuss.
And to understand the approach we ended up taking,
I think it helps to discuss
some of the most common constraints
that we face in data for journalism.
And how those did and didn't apply
to this particular project.
First are the related concepts of time and money.
Nobody has infinite time, but in journalism we tend
to have a little less time than in some other fields.
Even so, the FinCEN files was a bit of an outlier.
From the very beginning editors understood
that material deserved a long and thorough investigation.
And what about money?
Money as you might have heard is a bit tight
in the journalism industry.
Financial constraints don't typically have a huge effect
on the tools we use since our stack is almost entirely free
and open source, but money does have an effect
on human resources.
How many people we can afford to have working
on a particular project for instance.
Here too the FinCEN files was an outlier.
We had hundreds of people working on it.
Not all of them full-time of course,
especially during the pandemic,
but it was still a huge difference
from the typical investigation.
Aside from resources, there are constraints related
to the things we want to publish.
Perhaps the most important are specificity and accuracy
and the sort of tension between those two things.
Investigative journalists want to be very specific.
Rather than to say only in general terms
that there have been failures to combat money laundering,
journalists want to be able to say by whom
and when and how much.
When it comes to data analysis however,
this love for specificity is often at odds intention
with more cutting edge probabilistic techniques.
An algorithm that produces correct results
for 90% of inputs may be revolutionary in some fields,
but largely useless for many journalistic projects.
And finally one more constraint I'd like to mention,
something you might call explainability.
Can we explain our results to our audience?
And will they understand us?
Now, whether something is explainable
depends on the audience.
Internally among an audience of colleagues
there's a relatively high tolerance
for complex data analysis.
but for our external audience,
we're very sensitive to the complexity of what we publish.
Our readers are ideally a huge cross section of society.
And that includes people who don't know the meaning
or even scared by phrases like linear regression
or natural language processing.
Of course that doesn't stop us from using linear regressions
or natural language processing in our work.
But we want to make sure that whatever techniques we do use,
we can fairly and accurately describe them
in simple, plain language,
and that those results make sense to general readers.
We spend a lot of time thinking
about how exactly to do that.
So with all of this in mind, back to the narrative parts
of the suspicious activity reports.
Those narratives span 10,000 pages
and about three million words.
We wanted to identify the individual transactions
described within them.
At first we thought, being computer programmers,
maybe we can automate this.
Perhaps we could use regular expressions
or train an algorithm to identify the transactions
and extract the relevant variables,
the names of people, banks, payment amounts
and the relationship between those variables.
But we found these narratives were written
in such a wide variety of styles,
each bank employing a different structure and jargon.
We also noticed that some of the descriptions
might be fairly straightforward, but others depended
on nuanced readings of the full narrative.
The correct interpretation of a sentence for instance
might depend on a set of facts
described five paragraphs earlier.
After testing out a few automated approaches unsuccessfully,
we started to worry that we might spend
more time cleaning up and fixing those results
then we would actually save from the automation itself.
We did of course have another option.
To read each narrative and to enter all of the data by hand.
It sounded crazy at first, it certainly did to me,
and it would be a huge undertaking.
But the more we thought about it,
the more it started to make sense.
We believed it would be more accurate, which was great.
We also believe the process would be
much more understandable to readers, also great.
And the constraints of time and money
or human resources were less problematic
then in other projects.
And here's where the ICIJ collaboration
was incredibly helpful.
Thanks to this network of reporters
around the globe we were able to distribute the work
across more than 80 people.
Each reporter entered the data according
to a central set of rules.
A process led by ICIJ's data editor,
Emilia Díaz-Struck, and my colleague, John Templon.
ICIJ then built a custom Django app
to coordinate the verification
of this manually entered data.
ICIJ's team of experienced researchers
used the Django platform to review every entry three times,
making necessary adjustments along the way.
It was kind of like an in-house Mechanical Turk
but with secret documents and highly skilled reviewers.
The data from the narratives became an essential resource
for us and for our partners,
making a possible search in a structured way
for transactions that met only certain criteria.
This process provided another bonus.
By reading the documents very closely
for the purposes of accurate data entry,
reporters got a much better sense
of the documents themselves.
The data ended up being its own form
of research and reporting.
So even with the very same set of documents,
these tables and these narratives
and the same actual report,
we still needed two very different approaches
to extract the data we needed.
Computers where computers worked best
and humans where humans worked better.
Now I've been told that a theme of these conferences
and of this summit is the concept of data teams.
So I thought it'd be appropriate to conclude on this topic.
At Buzzfeed News data efforts are spread
across the newsroom.
The small team I run is technically part
of the investigations team,
but we also collaborate
with reporters in every corner of the newsroom.
And Buzzfeed News more generally
is a very collaborative organization.
There are people who are savvy in working with data
who work on other teams and likewise a breaking news team
might collaborate with a science team and so on.
And our best work on the data team has come
through collaborations with non-data reporters.
Even so I approached the ICIJ collaboration
with a bit of hesitation.
It's only natural.
After all it takes a lot of trust
to collaborate on a project this secretive
and this sensitive.
Over time however, not only did we learn
to trust ICIJ very much, but we came to treasure,
truly treasure our collaboration with ICIJ's data team,
Emilia Díaz-Struck, Augie Armendariz and others.
It was an amazing and unique experience.
We collaborated on the data extraction
and we were working from the same datasets,
but we did the analysis separately.
They had slightly different goals and constraints
and they were working on a different set of stories.
But we'd also come together to compare finance,
to critique each other's work and to learn from each other.
What one person had seen in the data
might be helpful to another person.
Having that additional perspective,
that sort of parallel set of eyes looking
at the same analysis, the same set of documents,
was invaluable.
I don't know the next time I'll get to participate
in another collaboration of this type and magnitude,
but the experience was enlightening and truly invigorating.
And has given me all sorts of thoughts
about how to improve our own processes.
And I look forward to watching the other speakers
and sessions at this conference
and learning more about how data teams
and collaborators work in other fields.
It's something I'd like to learn a lot about.
I think journalism has a lot to learn from other fields.
And so thank you for your time,
and I wish you all a fruitful conference.