Hi, I'm Brandon Stewart and I'm an assistant
professor in the sociology department.
I develop new techniques in an emerging
and interdisciplinary field called 'text
as data' - an approach for converting
collections of documents into data that
we can use to learn about the social world.
The study of human behavior is changing. In
fields like Sociology and Political Science,
it used to be that when we wanted to know what
people did or how they felt about something,
a central approach was to call up a random
set of people and give them a survey.
This was effective, but also expensive.
In the last two decades
things have been changing.
There's been an avalanche of new written
documents available: emails, tweets,
Reddit posts, press releases, and even
books have exploded in record numbers -
the space of the commons is changing.
These collections contain trillions
of words, more verbose than the book
collections at the world's largest libraries.
Social scientists realized that now
they had a new source of information of
what people were thinking and feeling:
the digital footprints that they left
as they lived their lives in print. That
is, if they could find a way to turn
those trillions of words into usable data.
Of course, the kind of public opinion
that we find in tweets or Facebook posts
is different from the kind of public
opinion we get from a telephone survey.
It actually harkens back to the pre-
telephone notion of public opinion,
not as the private opinions held by
the person picking up the phone, but
the opinions of those who chose to
proclaim their opinion in public.
And it isn't just public opinion,
many kinds of information of interest
to social scientists are encoded
in text. Being able to access it
isn't just a change in how we study
people, but who we study, and what we
study. The core ideas in text as data
have actually been around for a while.
Frederick Mosteller and David Wallace,
statisticians at Harvard University and
the University of Chicago, published
a paper in the journal of the American
Statistical Association back in 1963
entitled, Inference in an Authorship Problem.
They considered the Federalist Papers,
an unsigned collection of 85 documents
written by Alexander Hamilton, John Jay,
and James Madison about the U.S. Consititution.
While all the papers were unsigned at
the time of publication,
Hamilton and Madison provided
conflicting lists of authorship
before their death.
The result is that while the authorship of
73 of the papers is uncontested, for 12 of the
papers it was a matter of historical debate
if they were written by Hamilton or Madison.
A debate Mosteller and
Wallace set out to resolve.
They noticed that Madison and Hamilton
wrote in a subtly different style,
using filler words like upon, by,
from, and to at different rates.
They reasoned that if they could identify
the typical rate of usage of each of
these words from the papers with known
authorship, they could identify the authors
of the disputed essays based on whether
they use those words with the same rates.
Crucially, they didn't want to choose words
that conveyed the topic of the papers, but
rather those that reflected the unconscious
stylistic preferences of the authors.
At the time even counting the words proved
to be quite challenging. In his autobiography,
Mosteller described the process of
counting words by typing them on
long pieces of paper and then cutting
them into tiny one word strips.
When the counting was going on,
if someone opened the door, slips
of papers went flying around the room.
Once they had the counts complete,
they used statistical models that were
estimated on an IBM 7090, a then brand
new computer that filled a room but
which had far less computing power than
many people today have in their watches.
Once all the work was done, Mosteller
and Wallace were able to decisively
attribute the authorship of the disputed
papers to Madison, a conclusion that
matched the consensus of historians.
The computing resources of the early
1960's limited these kinds of approaches
to only relatively small corpora and
well-resourced researchers. Today, we can
scale up to even larger collections of documents.
In recent U.S. elections,
if you went to a rally or to visit the
website of a political candidate, you would
likely be invited to join an email list.
This is an interesting type of
communication because it's 'semi-public' -
anyone can sign up to receive a candidates
emails, but unlike television ads or
press releases, the emails are not
readily available to the public or
straightforward for academics to study.
During the 2020 U.S. election, I worked
with Arunesh Mathur, Arvind Narayanan
and a team of researchers at Princeton's
Center for Information Technology
Policy to create the largest collection
of election related emails available.
We used automated bots to visit
all the websites of thousands of
candidates for federal and state office
and sign up for their email list.
We then collected the emails over
the course of the campaign and released them -
all 435,000 of them
as the Princeton Corpus of Political
Emails. You can go have a look at them
yourself at electionemails2020.org
and search for your own representative
or your favorite candidate. With
hundreds of thousands of documents,
it isn't possible to read every one,
so we used statistical topic
models to help identify latent themes.
Topic models take a collection of
documents and return sets of words that
communicate a common subject matter
as well as the documents that contain
information about those subjects.
We can track, for example, the
way two parties talk about topics
like the COVID-19 pandemic, or
topics around racism and policing.
We didn't know that these
were the topics in the emails.
They emerged inductively
through our analysis.
In our case, we use something called the
Structural Topic Model, a tool I developed
with my colleagues, Molly Roberts at
UCSD and Dustin Tingley at Harvard. STM,
as we call it, works by using the
co-occurrence of words to figure out
the underlying topics of a document.
For example, if the words bat, pitcher,
catcher, base, and stadium all frequently
occur together in many documents, we might
infer that there's a common topic called, "baseball."
Every document is described then as
a combination of different topics,
so any given document might be 60% about
baseball and 40% about entertainment.
In the case of the emails, we learned
a number of different topics that
we grouped into categories like
political issues, fundraising,
voter mobilization or ceremonies and holidays.
As we went through the emails, we started
to notice something a little troubling.
A challenge for email senders is that
only about 20% of political emails are
even opened by recipients. In an attempt
to get people to open their emails,
senders were resorting to a
set of manipulative practices.
We decided to characterize and
measure some of these practices.
We grouped the things we found
into two broad categories.
One is clickbait.
That's using techniques such as
sensational headlines or urgent
deadlines to create interest. And two
are dark patterns, manipulations of
the email user interface to trick the recipient.
In these example emails you can see
the kinds of things we're talking about.
Changing the name of the From Field
so it isn't clear who it's from, or
placing "Re:" in the subject
line to make it appear as though the
email was a reply to a previous message.
We trained a statistical model to measure
these tactics in the email corpus and
found that they are distressingly common.
The median active sender uses them in 42% of
their emails and 99% of the senders use them
at least occasionally. We
collected over 400,000 emails -
this would be way too much
to go through manually
and look for this information.
Text as data methods both helped us
learn what the emails were about,
read them more effectively, and
characterize the rates at which
manipulative tactics were used.
Text as data methods are really flexible:
they work across languages and can even be
used help make causal inferences. Several
years ago, Molly Roberts, Rich Nielsen
and I wanted to study censorship in China.
It's been well-documented that the
Chinese government censors speech on the
internet by taking down individual blog posts.
But censoring the entire
internet is really difficult,
the number of bloggers is always going to
be much greater than the number of censors.
This naturally leads us to wonder
about the role that self-censorship
might play in the system -
that is if a blogger is censored at one time,
does it change what they write in the future.
This is a difficult
causal inference question:
people aren't censored at random
so if we simply compared the writing of
people who were censored to people who
weren't, we would conflate the effect of
being censored with the kinds of people
who were more likely to write the sort of
posts that got censored in the first place.
One common strategy in causal inference
is to imagine the ideal experiment, even
if we can't practically do it and then
try to emulate it with observational data.
In such an ideal experiment,
we would randomly assign censorship
to some posts and not to others and
then follow those bloggers over time.
Of course, this has the double
problem of being wildly impractical
and completely unethical.
But what we do instead is try to create
the best approximation we can using
real data. Using data from 2012 we find
pairs of posts written by similar users,
with similar censorship histories, who on the
same day write posts with similar content,
but one was censored and one was not.
That is, we comb through the
data to find censorship mistakes.
We then compare the outcomes
of these pairs of bloggers
and we look at over 75,000 posts using text
as data methods to find these mistakes.
Ultimately, we find that the users who
are censored have almost doubled the
censorship rate of those who weren't in
their post moving forward, but they still
posted the same rate they did before.
This suggests that censored bloggers
are undeterred by censorship and either
still posting on sensitive content or
possibly subjected to more intense scrutiny.
Text as data methods are reshaping
many fields of study from sociology,
to political science to medicine.
A key part of this is creating open
source software that helps other
scholars make use of these methods.
The STM software, the topic model I
mentioned earlier, has been downloaded
hundreds of thousands of times and
used by academics, businesses and
government agencies to help make sense
of the documents of countless communities.
This has expanded the range of who we study
and what we study and how we study it.
These issues, of course,
are not without challenges.
The ethics of studying large text corpora
can be really nuanced. While the things
people post on the internet are public,
the authors may not realize or consent
to their writing being used for
academic research creating questions
of what is shared in the common space.
The ethical considerations of
this kind of work are an active
discussion across numerous fields.
If you're interested in learning more about
this topic, Justin Grimmer, Molly Roberts, and I
have a new book from Princeton University
Press available in March of 2022 called,
Text as Data, A New Framework for
Machine Learning in the Social Sciences.
This is just the beginning of how these
methods will change the way we study people.