hi there I'm Zivvy Epstein and in this talk I'm going to explore how we can use
layperson judgments to combat misinformation on social media by down
ranking distrusted sources and hopefully shed some light on the
question will the crowd gained the algorithm this is joint work with Gord
Pennycook and Dave Rand. the rise of misinformation in social media has not
only posed a serious issue for democracy in the information age but also has
raised interesting social and technical questions about how platforms should
fight misinformation. Here's a taste of three possible approaches to this
question. The first line of defense is automated detection with algorithmic and
machine learning approaches but this has some problems. For one there's no clear
definition of what content and features should be including in training sets, and
relatedly is the fact that misinformation is non-stationary
and tends to evolve very rapidly. Finally there are thorny ethical questions about
how to treat false positives and when it's okay to censor content that an
algorithm tags as bad. A second approach is to have professional fact checkers
evaluate content to determine its veracity but this approach isn't
scalable because often the fact checkers can't keep up with how fast bad actors
can create false content and the evaluation process itself can be pretty
slow so the problematic content never winds up getting identified and even the
content the moderators do flag was probably flagged way after the period of
peak virality. In addition the use of professional fact checkers may actually
promote the acceptance of misinformation via the implied truth effect whereby
people interpret the absence of a warning as evidence that the story may
have been fact checked and validated, and finally the moderators themselves may
become traumatized or start to believe the conspiracies that they see all day.
A third approach involves the development of tools for
helping users detect misinformation for themselves. These can help individuals
in the sometimes daunting tasks of increasing discernment online. A major
limitation of these tools however is that they require people to opt into
using them which is a killer blow because people who are most susceptible
to this information may be less likely to choose to use such tools in the first
place. A promising alternative to these approaches that has received considerable
attention is to use crowdsourcing to identify misinformation as inputs to
a ranking algorithm. Here's how it works. First, the platform could illicit user
trust judgments about content, this could then be fed directly as inputs to a
news feed algorithm. The news feed would then downrank mistrusted content thus
preventing its spread. We focus on a system in which users judge the trustworthiness
of domains that produce misinformation rather than individual pieces of content.
This crowdsourcing approach is promising but the question remains: will the crowd
gain the algorithm to advance their own partisan agendas? there is some evidence
that layperson trusts may be distorted by politically motivated reasoning but a
growing body of evidence suggests that reasoning may not be held captive by
ideology when evaluating the accuracy of news. Survey studies find that people who
engage in reasoning are less likely to believe false political headlines
regardless of ideological alignment but even if their judgments are not impaired
by partisan bias. There are some evidence that laypeople may be simply unequipped
to identify misinformation due to a lack of media literacy. This suggests that
layperson ratings may be noisy and ineffective. In counterpoint to this
concern however is the large literature on the wisdom of the crowds which shows
that aggregating responses can dramatically reduce noise, and perhaps
most relevant here even if laypeople do produce effective ratings they may
choose not to in an effort to game or astroturf the crowdsourcing system
towards partisan ads but ultimately it's an open empirical question
if they would do so. These arguments and counter-arguments underscore the fact
that although this crowdsourcing approach is fundamentally algorithmic,
the challenges that must be overcome in order to implement it successfully are
social in nature rather than technical and thus involve empirical questions
about how people would interact with such a system. In the present work we
first demonstrate that layperson trust ratings do indeed effectively identify
misinformation outlets by replicating the recent study. Second we investigate
if participants change their responses when they are told the results will be
used to inform social media ranking algorithms. Ee recruited participants for
a survey using Lucid, a survey platform which uses quota sampling to provide a
sample that is nationally representative on age gender ethnicity and geographic
region. 984 participants completed the survey and were indeed evenly
distributed across age gender ethnicity and political leaning. Each participant
was shown a list of website domains and was asked: "do you recognize the following
website" and "how much do you trust each of these domains." The domains were
randomly sampled from a set of 89 news web sites across the right-left
political spectrum that fall into three categories. First is mainstream media
outlets such as CNN or Fox News. Next is hyper partisan sites that produce
misleading coverage of events that did actually occur such as Breitbart and
Daily Kos, and finally we have fake news sites that generate mostly blatant false
content such as world news daily report and daily headlines. Each participant was
shown 10 mainstream sources 10 hyper-partisan sources and 10 fake news
sources, this allows us to compare their trust ratings of mainstream sources to
their trust ratings of hyper-partisan in fake news sources in order to construct
a measure of how discerning the ratings are and thus their ability to
effectively identify misinformation sites. To provide a firmer ground truth
we compared our participant's trust scores with trust ratings
collected from eight professional fact-checkers for a subset of the 60 of
the domains. We included a between subject design in which participants
were randomly assigned to either a control condition or a knowledge
treatment where they were told that their responses would be used to inform
at the ranking algorithms. Specifically they were told the overall results of
the study but not any individual's response will be used to determine which
news sources are relatively trustworthy and will be shared with Facebook with
the goal of improving their platform in particular the goal is for content from
sites which receive high trust ratings to be shown to more Facebook users and
content for sites which received low trust ratings this experimental design
allows us to causally assess the impact of this knowledge on discernment.
Now for some results. On the x-axis I have the trust among the Democrat
participants and the y-axis I have the trust among the Republican participants. Web
sites above the dotted line are websites that Republicans trust more than
Democrats and websites below the dotted line our web site that Democrats trust
more than Republicans. I also have two plots one for the control group and the
other for the group that was told that their responses would influence the
ranking algorithm. These plots tell a couple stories. First we see that both
Democrats and Republicans are pretty good at discerning mainstream domains
from fake or hyper partisan ones. We also see that there isn't so much a
difference between the control and treatment groups. taken together these
results suggest that people trust mainstream sources more than
hyper-partisan or fake news sources even when informed that their responses would
influence the ranking algorithm. This visual impression is confirmed by
entering trust ratings into a regression with the following independent variables:
condition, source type, and the interaction between the two. We see a
significant positive effects of source type such that mainstream sources
received higher trust ratings than non mainstream sources and no significant
main effect of condition nor significant interaction between source
type and condition. This implies that knowing that the ratings will inform
ranking algorithms had no significant impact on average trust ratings.
Next, we look at the differential effect of partisanship on the treatment by
conducting the same analysis but also including a dummy for a participant
partisanship and all interaction terms. We find no significant three-way
interactions between partisanship source type and condition.
This suggests that knowing that the ratings will impact the newsfeed does
not affect how discerning people's trust ratings are for supporters of either
party. Consistent with prior work we do see a significant two-way interaction
between partisanship and source type such that Republicans trust mainstream
sources less than Democrats. This is not to say however that the knowledge
treatment had no effect although the treatment did not affect the crowds
ability to effectively discern between mainstream and misinformation sources. We
did see an increase in political polarization. We defined polarization as
the absolute difference in trust ratings between Democrats and Republicans which
is visually represented as the degree of dispersion away from the diagonal
dotted line. Here is the distribution of overall
polarization in the control of versus treatment. This suggests that our
treatment made people respond in a more partisan fashion but because people's
polarization was small and pretty symmetric across party lines they seemed
to cancel out when computing overall discernment. Therefore the crowd ratings
still effectively identify misinformation sources. Finally we
compare these results to ground truth by calculating the correlation between the
politically balanced layperson ratings and the fact tracker ratings. These are
very high in both conditions: 0.87 for control and 0.88 for treatment. and not
significantly different from each other as determined by a Fisher R-to-Z
transformation. Thus the judgments of the laypeople in the treatment are just as
highly aligned as those of the professional fact checkers as the lay
people in the control. For completeness we repeat the same comparison with the
fact checkers considering Democrats and Republicans separately. In both cases
as we find no difference across condition although we do find the
correlation with the fact checkers is significantly lower for Republicans
compared to Democrats. To wrap up we are super excited about crowdsourcing trust
in down ranking untrusted outlets as an approach to fighting misinformation. We
found that late person trust ratings are effective in discerning between high and
low quality news outlets. Rather than being blinded by partisanship our
participants trusted mainstream sources much more than hyper partisan or fake
news sources. We also found that informing people that their responses
will influence ranking algorithms did not affect discernment. This knowledge
did increase the polarization of their responses but cancels out in the end. We
also found that Republicans were less discerning than Democrats but
interestingly this was not because Republicans trusted misinformation more
but rather because they trusted mainstream sources less. This adds
another piece of evidence to debates of ideological asymmetries in media
judgments. Building better media environments requires a bunch of future
work. It's important to note that our framework uses a web site level trust
score which does not take into account the variation in article quality that
each web site publishes. Relatedly our method only considers eighty-nine
websites and the american media ecosystem. Understanding how these
results generalized to different media cultures is crucial before such a system
could be applied on global social media platforms, and finally and perhaps most
importantly it's clear a crowdsourcing approach must consider whose opinion is
being asked. How do we decide who decides? Answering this question requires not
only transparency into content moderation but also radical innovation
in how platforms are governed. Thanks for listening!