Hello, everyone.
I'm Prerna Juneja from University of Washington.
I'm here to present our paper where we investigate
misinformation that is algorithmically
curated by YouTube platform.
In the past few years, several journalistic reports
have accused YouTube's algorithm of driving people
into the internet's darkest corners.
So you see opinion pieces like this
in the press that call YouTube a great radicalizer.
And in response to these accusations,
last year YouTube said that it would stop recommending
fewer misinformative videos.
But all these journalistic reports
and even YouTube's response [INAUDIBLE] were anecdotal.
And that there is no enough empirical evidence
to prove any of these claims or to disprove YouTube's counter
response.
So the aim of our study was to verify these anecdotal claims.
That is, we wanted to determine whether YouTube
surfaces problematic content in its searches
and recommendations.
And if so, then under what conditions?
So we verified these claims by performing systematic audit
experiments on their platform.
And we considered one particular type of problematic content.
That is, conspiracy theories.
So these are the three research questions
that our study answers.
In the first research question, we
ask whether user's demographics like age and gender
or user's location have any effect
on the amount of misinformation that is presented to users.
In the next question, we investigate the effect
of user's watch history.
That is, what if user builds history
by watching videos that are all pro-conspiracy, or all neutral,
or all debunking, what happens then?
In the third research question, we
want to compare the topics like for which
topic does YouTube return most misinformative content?
So this is our audit pipeline.
We started by selecting five conspirational topics that
are high impact.
That is, they're popular, perennial,
and demonstrably false.
Next, we curated search queries for all the selected topics
using Google Trends and YouTube's Autocomplete
Suggestions.
And we audited three components of YouTube, the Up Next
Video, the Top Five Recommendations,
and YouTube's Search Results.
Now coming back to the first research question,
to test the effect of demographics,
we considered four different age groups and binary gender
and created eight accounts with these combinations.
And to study the effect of geolocation,
we determined hot and cold locations
for each of the five topics.
Hot regions are the states for the topic was most searched.
And cold regions are the states for the topic
was least searched.
For example, the Flat Earth topic
was most searched in Montana and least searched in New Jersey.
And so we created two accounts for each topic that were
drawn from different locations.
So once we have the search queries,
we can launch our audit experiments.
We conducted two sets of audits, search and watch,
each consisting of two experiments.
In the search audit, we investigated YouTube's search
algorithm and created brand new accounts
that had no history whatsoever.
[? Whereas ?] in watch audit, the account
had built some history by watching
videos use of certain stance.
Here we investigated both YouTube's search
as well as recommendation algorithm.
A byproduct of these audits for around 3,000 unique videos that
were manually annotated for the stance on the conspiracy
theories.
Not as whether the video promotes, has a neutral stance,
or debunks the conspiratorial topic.
And then we performed statistical tests
on the annotated data.
Now let's discuss the results of the statistical test.
In the first research question, we
found that for brand new accounts,
demographic and geolocation do not have an effect.
However, once account builds a history,
both demographics and geolocation
exert an effect on recommendations,
but for certain combinations of topics, stance, and company.
So for example, males watching neutral videos
on certain topics received more conspiratorial recommendations
than female accounts.
For the second research question,
we found that watch history had a significant effect only
for search results of vaccine controversy topic.
We found evidence of a filter bubble effect
where accounts watching anti-vaccination videos,
received more such videos in their search results.
Watch history also has a significant effect
on the recommendations.
And here is the interesting fact.
Recall I just told you that if you watch anti-vaccine videos,
you'll get anti-vaccine videos in search results.
However, if you watch anti-vaccine videos,
YouTube actually recommends debunking videos in the Up Next
and Top Five components.
In the third research question that we
wanted to determine the topic wise difference,
we found that chemicals topic had significantly more
misinformative search results than other topics.
On the other hand, 9/11 conspiracy topics
had significantly more most informative recommendations
than other topics.
Now what are the takeaways and contributions of our work?
So first we developed the methodology
to audit search engines for misinformation, which we hope
will be emulated and applied to other platforms.
We have released a data set of 56,000 videos linked
to the stance of video and the personalization attribute
audited.
And through our audits we were also
able to identify vulnerable populations who
were presented more misinformation than others.
For example, we discussed how men
were recommended more misinformative videos
than females.
So one of the implications of our findings
is that YouTube might be reacting to certain media
reports.
It seems that it is handpicking topics and modifying
its algorithms selectively.
So now what's happening is that while YouTube is recommending
debunking videos to those who are watching
anti-vaccine content, the same thing
does not happen when users search
for vaccines on the platform.
So finally to conclude by what we started with,
we find that YouTube has started modifying
some of its algorithms for certain topics.
But it has a long way to go before it
can mitigate the misinformation problem from its platform.
That's all I had.
Thank you very much.