My name is Waheeb Yaqub.
I am a Ph.D. student at The University of Sydney.
I am presenting our paper titled “Effects of Credibility Indicators on Social Media
News Sharing intent.”
Online misinformation, often commonly referred to as fake news, has grown rapidly in the
past few years.
The reach and impact of such content is amplified because people sharing it with each other
via social media platforms.
One technique to overcome and limit the sharing of misinformation is by labels and annotations
that indicate the credibility of a piece of information to help user separate true and
false information.
Such visual and textual cues are what we term as credibility indicators.
Such indicators are sometimes incorporated to mark some of the content on social media
platforms like Facebook or Twitter as shown in these examples.
On the left you can see a Facebook warning legally required in Singapore for marking
content with false information.
The middle example shows Twitter pointing users to related factual information that
can counter misinformation.
On the right is the disputed information flag used previously by Facebook before it was
discontinued.
The hope is that the presence of the indicators will dissuade users from consuming and subsequently
sharing questionable content.
Credibility indicators can be considered as substitutes for other parties that people
might typically consult to judge reliability and trustworthiness of information.
We considered four such sources:
Domain experts, News outlets, Other people in the community, and technological systems.
We mapped these sources to four corresponding entities:
professional fact checkers, news media, the public, and artificial intelligence techniques,
respectively.
We then created corresponding textual credibility indicators for each of these entities.
As you can see, the text for each indicator was identical except for the entity that was
referred to as the underlying source.
Based on these indicators, we conducted a study to tackle three specific research questions.
First, whether attaching credibility indicators to news headlines would reduce the intention
to share the content on social media?
Second, whether the different types of credibility indicators vary in their influence?
And third, whether the influence of the credibility indicators varies based on demographics and
other factors?
To investigate these questions via a study, we first collected a carefully selected set
of 72 news headlines from well-known as well as obscure sources.
The collection of headlines covered a variety of topics, and each headline was connected
to an article that was either true, false, or satirical.
More details about the collection, including our selection process, are in the paper and
supplementary materials.
Each participant in our study was shown 12 randomly selected headlines from the larger
set of 72.
These 12 headlines were composed of 4 true and 8 non-true articles.
Participants were randomly allocated to control or treatment conditions.
In the control condition, the headlines were presented without any credibility indicators.
There were four treatment conditions, one corresponding to each of the credibility indicators
mentioned earlier.
In the treatment conditions, any non-true headline was presented with the corresponding
credibility indicator shown under the headline.
Here is an example headline used in the study for a false article from an obscure source.
In the control condition, participants saw just the headline whereas in the treatment
conditions, participants were shown the credibility indicator corresponding to the condition.
This example shows the indicator used in the Fact Checkers conditions.
For each presented headline, we asked participants whether they would share it via social media.
The study was conducted via the Amazon Mechanical Turk crowd work platform and consisted of
the following steps.
We published the study on Amazon Mechanical Turk.
The study was open to anyone from the US with more than 95% approval rating for prior tasks.
After consenting to participate, participants were randomly assigned to one of the study
conditions described before.
In each condition, we collected the sharing intent for 12 randomly selected headlines
as explained previously.
Next, we echoed the answers back to the participants and asked for the reasons behind each of the
12 choices.
Additionally, we collected demographics and other relevant information.
After filtering out those who failed attention and manipulation checks, we received 1,512
valid responses from individuals all across the United States.
We used binomial logistic regression on this data to answer our research questions.
We found that the presence of a credibility indicator of any type did indeed reduce the
intent for sharing non-true headlines.
Participants in the treatment conditions showed a reduction in sharing intent for non-true
headlines by 2.5 compared to those in the control condition.
Moreover, in more than half of the cases, participants in the treatment conditions explicitly
indicated using the credibility indicator when deciding whether they would share the
content on social media.
Only about 1/3rd of the participants indicated not using the indicator when deciding while
the remaining 11% based their decisions on other factors.
However, the indicators were not equal in their influence.
When comparing across the treatment conditions, we found that their impact varied.
We found that the Fact Checkers indicator was the most effective among all, leading
to a reduction in sharing intent for false content nearly 5 times compared to the control
condition.
This was confirmed independently by the participants in their ratings for the various indicators
that we collected at the end of the study.
Participants rated the Fact Checkers indicator as the most effective, regardless of study
condition.
Interestingly, the AI indicator was rated the least useful even though AI techniques
are often presented as lacking the bias and partiality of human based approaches.
We discovered that the influence of the indicators differed based on a person’s political affiliation.
As we can see in the figure, for each of the four indicators, the largest decreases in
the intent to share non-true content were observed for the Democrats.
The corresponding decreases for Republicans were much smaller.
In fact, for the AI indicator, Republicans indicated the intent to share false content
more than in the control condition.
The influence on Independents fell somewhere between that for Democrats and Republicans.
The influence of the indicators varied by gender as well.
For three of the four indicators,
women participants were influenced to a greater extent compared to men.
However, the differences between genders are comparatively smaller than those based on
political affiliation.
In addition to the sharing intent, we examined the reasons participants chose for not intending
to share false headlines via social media.
In the control as well as treatment conditions, a notable proportion of the participants intended
not to share non-true headlines because of their false nature.
This reason was provided 7% more in the treatment conditions than in the control condition,
suggesting that the credibility indicators provided a boost to credibility judgments.
For 30% of the non-true headlines, participants in the treatment conditions marked that ..they
were influenced by the credibility indicator.
Our results suggest that embedding credibility indicators within social media platforms can
help curb the spread of misinformation via these systems.
Since the influence of the indicators varies by type, we call for a broader exploration
of the visual and textual design space for generating effective indicators that can have
a large impact on reducing misinformation sharing.
The personalization capabilities of social media can further help customize the indicators
such that they are tailored based on the kind of indicator that is likely to be the most
persuasive for each individual user.
Importantly, our findings highlight that experts, such as fact checkers and journalists, must
be
an integral part of the pipeline through which content is distributed on social sharing platforms
that can spread misinformation in a viral manner.
We need to design tools and processes that can help raise the effectiveness of fact checking
at scale.
Thank you very much for your attention.