Today I'll be talking about my paper "Fake News on Facebook and Twitter": Investigating
How People (Don't) Investigate.
And this is work I've done with Savanna Yee and my advisor Franzi Roesner.
So misinformation which is the false information that is spread regardless of if there's an
intent to mislead, has gotten a lot of press coverage lately.
It was Dictionary.com's word of the year a couple years ago.
And rightly so.
It's had a lot of impacts on elections, public health, and public safety.
And this has been a prevalent issue, especially on social media, because 67% of Americans
use social media to get some of their news.
And these platforms have taken various steps to try to combat these issues, particularly
with design affordances.
So Facebook has some "Fact-Checker" labeling for articles, as well as a button to give
information on a news source's background and history.
And Twitter has banners that provide more information related to health.
And I should say that this study was done a year ago, so these platforms have updated
the methods they use to intervene on these issues.
So there's a lot of work in this space, particularly around studying what kind of cues people use
to determine if information is credible.
There's also work on investigating how people consume news on social media and what kind
of strategies they use to determine if its credible.
And other work has explored the effects of how these cues actually determine whether
or not someone reshares news.
However, what a lot of this work does is either presents misinformation through a researcher-created
account that the participant is instructed to follow, rather than seeing misinformation
posted by they know.
Or these studies particularly instruct participants to look for fake news and talk about why they
think something is misinformation.
So this raises the question for us as researchers of how can we get a more natural understanding
of how people interact with misinformation posts on their own feed?
To do this, we conducted a 3-part deception study.
The first part of the study, we conducted semi-structured interviews about their social
media usage.
We asked questions like "who do you follow" and "what kind of content are you interested
in seeing"?
During the second section, we had a think-aloud session where while participants scrolled
through their feeds, they were instructed to talk about how they were interacting with
content on their feeds, with a researcher in the room.
And they did this while their feed's appearance was changed by our browser extension in order
for misinformation to show up on it.
And finally, we conducted exit interviews where we asked about what kinds of past experiences
people had with misinformation and how they go about investigating social media posts.
And then this was followed by a debrief where we told participants about the true nature
of this study.
So to clarify on what the extension does, we developed it so that misinformation content
would be overlayed onto existing posts on a person's feed, so it would appear as if
someone the participant actually follows was posting this information.
We found this content through Snopes, and these topics covered politics, health, msic.
And the post types we found were a mix of articles, memes, text, and photos.
And we asked people to talk aloud about why they were pay attention to a certain post,
or skipping it, etc., etc.
And we want to stress that participants couldn't actually share misinformation on their feed
or offline.
And here's an example tweet that a participant might see as if posted by someone they follow,
and that's an example of an article that would show up on Facebook.
We recruited adults who used Facebook or Twitter daily or weekly for news.
And we had a total of 25 participants.
They were all either left-leaning or independent.
That's one of the limitations of our work.
And given the deceptive nature of our study, we thought a lot about ethical considerations
to minimize potential harms.
So participants could exit the study at any point and retain compensation, and during
the debrief, participants were shown screenshots of misinformation posts that showed up on
their feeds.
To minimize loss of self-esteem and normalize reactions to false news, we told participants
that identifying misinformation is difficult and it is common to not realize its fake.
And participants didn't show any outward signs of distress.
And this study was approved by the UW's IRB.
To analyze the data, we did iterative coding with 2 researchers, who coded inductively.
Each coded half of the interviews and double-checked the other's coding for consistency.
Onto the results.
So we documented a variety of response to how people interacted with misinformation
on their feed.
I'm just going to focus on a couple parts of it.
When someone saw posts sometimes they would choose to ignore it for various reasons, such
as the content not being engaging.
For example, when P10 said on skipping an article: "If it was something
funny like a meme or something, then I'd probably care about it, but it's just words.
So I'm a little bit less interested."
Some of these reasons were also similar to reasons why people would choose not to investigate
a post, even if they were skeptical.
So p18 said, about how long they'd spend on a tweet before moving on, "Probably less than
8 seconds."
Some people did choose to investigate posts when they were skeptical.
And there were various ways they would do this, from investigating claims by googling
for sources, or by evaluating by the person who posted it or or the news source of the
actual post.
P24 said on investigating the news source, "If it's like Al and Bob's website, I'm not
going to click on it.
If it's like CNN Bob or CNN South or something, I'm not going to click on it because they
put CNN in it to make it seem factual where it isn't.
But if it's like cnn.com, then I'll click on it."
And P24 mentioned that he had learned this the hard way.
He started checking for news sources after he believed a click-baitey headline that said
a basketball player had been traded, but then ESPN disabused him of that notion.
I also want to mention that while there were various affordances within social media that
people did use to investigate posts such as hovering over an icon to get context on the
background of who the person is posting something, or looking at the comments of a tweet or Facebook
article to see if anyone has said "This is untrue", things like the "i" button on Facebook
didn't get mentioned at all amongst our participants.
So this button is basically used to show when an organization was founded and other contextual
information to help people determine if something is credible, but among our participants, no
one had said they used it, and most had said they had never noticed it at all.
This raises the question of the utility of these affordances if they're not large enough
or visible enough for people to react to.
Now I will discuss some of our findings in terms of credibility heuristics, social media
mental models, and future work in this space.
Our work supports theory on how people use "shortcut" cues when they're not highly motivated.
Oftentimes when people don't find information that's relevant to them or they don't have
enough time to put attention into really analyzing some information, they use sort of shortcut
cues or heuristics in order to determine whether they believe it or not.
And these range from using the news source or relying on their trust in the poster, or
finding multiple sources so they would triangulate information, trying to determine if this is
typical content coming from that poster, or if they thought misinformation looked like
an ad they would skip it entirely.
I do want to mention that some of our participants self-reported during the exit interview that
they would use the news source to evaluate information, during the think aloud session
only P24 did so.
No one else checked the news source URL.
And most tended to rely on trust in the poster, rather than trust in the news source.
We also found that many of our participants had surprisingly good mental models of their
feeds.
THey had very strong ideas of what kind of content someone that they follow would post.
And they were fairly aware when someone, when we had inserted misinformation into someone's
post and it seemed out of place.
And we find that this is an opportunity for social media platforms to design for debunking
fake news for these types of attentive users.
We also found that some of our participants follow news sites that they don't agree with
their ideology or content.
And this contrasts with the idea that social media websites form ideological echo chambers
where people only follow stuff they believe in.
It didn't seem like any of our participants knew about the Facebook "i" button.
But Twitter and Facebook have recently rolled out new misinformation interventions since
this study, and while we didn't study any of them in this project, it's work that my
team is currently looking at.
We have some thoughts on where future designs in this space of misinformation interventions
could go.
Affordances could be designed to save suspicious posts for later if the user doesn't have time
to investigate in the moment or if they're on their phone and they find it difficult
to investigate a post.
And our work also raises future research questions.
UX for fact-checking may be most effective for targeting "non-wary" information consumers,
rather than people who are extremely skeptical of institutions and social media websites.
And this raises the question of what types of information consumers affect how interventions
work for them?
Thanks for coming to my talk and if you have any more questions about the variety of ways
we've documented how people interact with misinformation, feel free to read my paper
or ask me questions.
Thanks!