Hi, I'm Jennifer Allen, and I'm a third year PhD student at the MIT Sloan School of Management.
Today, I’ll be presenting a paper titled “Birds of a Feather Don’t Fact-check Each
Other: Partisanship and the evaluation of news in Twitter’s Birdwatch crowdsourced
fact-checking program So as background for this project, in January
2021, Twitter introduced birdwatch its community-based approach to misinformation and invited users
to apply to participate in the pilot program.
And so the basic high level goal behind the program was to create a community where ordinary
Twitter users could apply to join Birdwatch, and then once they were accepted by Twitter,
help identify misleading tweets on the platform.
And the way the system works is first Birdwatch users flag, any tweets that they come across,
you know, either in their feed or by seeking out in some way, and then submit a fact check
note labeling the tweet as either "potentially misleading," or "not misleading"
So in this example on the right.
A birdwatcher has flagged the tweet on the top as potentially misleading, and then written
a short explanation justifying their reasoning for calling this tweet misleading.
And then the second important component of the way that Birdwatch works is these "ratings."
So these work by birdwatchers navigating to a separate Birdwatch site, where they are
shown notes written by other Birdwatchers and
asked to rate them as either helpful or not
helpful.
So an example of what this looks like is on the bottom right here.
You know, this is an example of what the Birdwatch site looks like.
And then Twitter actually aggregates all these helpfulness up votes or down votes and
highlights some notes as "overall rated helpful" by the Birdwatch community and calls them
out and highlights them on the Birdwatch-- the Birdwatch
site.
So it kind of functions similar to Reddit in that way.
So, naturally once the Twitter-- once Twitter put out this call for Birdwatchers and introduced
this platform, there was a lot of skepticism by the community.
So here's some replies by people in the community about the introduction of Birdwatch, and my
personal favorite, and which kind of exemplifies the
rest of the replies is, "Thank you, Twitter, I'm looking forward to abusing this feature."
And I think these replies and response that Twitter got really gets to the heart of questions
that we as researchers had as well.
So you know, obviously, misinformation on Twitter is a problem.
In an ideal world Twitter clearly hopes that empowering users to identify that misinformation,
and help identify that misinformation, could function something like the picture on the
left, you know, as some sort of reasoned debate, or marketplace of ideas, where the truth wins
out.
However, social media is not an ideal world as we well know.
And as the response to the announcement exemplifies there's a really genuine worry that people
will not engage in this opt-in platform in good faith, and it'll function more like the
picture on the right, where people call political
content from across the aisle that they don't agree with fake news.
And so that's the question that we as researchers were really focused on in this work and really
focused on understanding about the Birdwatch platform is, how does political partisanship
shape users' participation in Birdwatch.
So, you know, to what extent is it important-- is political partisanship
important-- in the tweets that Birdwatchers choose to flag as misleading, and in the
fact check notes that they choose to rate as either helpful or not helpful.
So now moving on to the actual data sets that we worked with for this project.
So, the first important data set is the data set of tweets that were actually fact-checked
by birdwatch users.
So important to note here that topic models showed that most of these
tweets were either political or about Covid, and in fact the top accounts
were AOC and Marjorie Taylor Green, who are Democratic and Republican Congresswomen respectively.
And so a lot of political content being a fact checked on this platform.
And so we were actually able to infer the partisanship of tweet authors as well, beyond
just looking at top accounts, using the method in Barbera at al, and looking at the accounts
that the Twitter -- or that the, you know, original focal users-- followed
and found that there was a pretty even split between Twitter tweets being written by Democrats
versus Republicans in the kinds of things that Bird watchers were fact checking.
So now looking at the fact check note data set, we had about 5000 notes by 1100 birdwatchers,
the vast majority of which were actually labeling the focal tweet as misleading.
And so, it's clear that this system is functioning as a sort of flagging system for tweets that
Bird watchers are identifying as potentially misleading.
For the ratings data set that we had about 30,000 ratings submitted by about 2300 bird
watchers-- about five ratings per tweet-- and
here we have a more even split between the
notes that were rated helpful versus not helpful, about two thirds of notes, being
rated as helpful by their fellow bird watchers.
And, also an important thing to note here between all these data sets, is that again
we had a pretty even split between participation by Bird watchers between predicted Democrats
and predicted Republicans, and the same for the tweets that were being checked.
So now, turning to the results and trying to understand how political partisanship is
really influencing participation on this platform.
So the first thing we looked at is how partisanship of the birdwatcher and the tweeter affected
the tweets that these birdwatchers were choosing to fact-check.
So the y axis here we have inferred partisanship score of the tweeter and on the x axis, we
have the political score of the birdwatcher, and so higher scores here correspond to more
Republican and lower scores correspond to more Democrat, and each point
corresponds to a note submitted by a birdwatcher.
Red points are notes where the bird watcher said the tweet was misleading, and blue points
are those that the birdwatcher said were not misleading.
And so the first thing that jumps out is that Bird watchers are much more likely to fact
check tweets by their counter partisans rather than tweets by their co partisans.
So Democrats are three times more likely to fact-check tweets by Republicans, Republicans
are one and a half to two times more likely to fact-check tweets by Democrats.
However, when they do flag their co partisan tweets, they're much more likely to flag them
as not misleading than when they flag their counter partisan tweets.
And so we're seeing evidence of partisan herding, and people choosing to police and call out
misinformation, or tweets that they think are misleading, by the other side to an overwhelming
degree.
So now turning to the ratings data, we see a similar phenomenon.
The bird watchers are more likely to rate notes by counterpartisans as not helpful and
rate notes by co partisans as helpful, again to an overwhelming degree.
So we're seeing this evidence, again, of partisan herding where people are cheerleading their
own team.
And then, you know, sort of down ranking and brigading the opposing team.
So looking at the these results, you know, is it reasonable to conclude that Bird
watch is doomed?
You know, is this partisan herding that we're seeing evidence that each side is calling
the other side fake news like we saw in that earlier picture.
And I think our initial preliminary results suggest you know maybe not so fast.
And so, to evaluate this question, we actually had two experts fact-check 111 of the tweets
and actually found that there was a high level of agreement between the tweets that
the fact checkers said were misleading, and the tweets that the most helpful Birdwatch
note
said was misleading.
And so you know, there's about 82% agreement between the bird watchers and the fact checkers,
and thus, it doesn't look like bird watchers are necessarily indiscriminately calling content
by the other side fake news, and it looks like it's possible
that bird watchers are actually policing misleading content written by the other side, although
of course more research is needed to be done.
And just some final thoughts just to sum up, kind of, the phenomenon that we're seeing
here
with this project.
First, as is obvious, you know, shared partisanship is a big predictor about which tweets birdwatchers
are labeling misleading and which fact check notes they're rating as helpful.
And, you know, with this in mind, it seems like partisanship and the desire to scrutinize
the other party's content is really serving as an important motivator, which is driving
people to participate in birdwatch-- this opt in platform-- in the first place.
However, despite that, you know, most tweets that are getting flagged by birdwatchers are
actually also marked as misleading by fact-checkers as well.
And so it's too soon to conclude that, you know, birdwatch is a failure or that it's
success, based on the partisanship herding that we're seeing.
So it's possible that people are just calling out misinformation by the other side, and
cheerleading their own side, but not in a way that sacrifices accuracy, although as
I mentioned earlier those results are preliminary, and more research is needed to be done.
And with that, I'll sum up -- looking forward to the conference.