Hi everyone.
My name is Aafaq Sabir and today I will be presenting my work on “Analyzing the impact and accuracy
of Facebook activity on Facebook’s Ad-interest inference process”
co-authored by Evan Lafontaine and my advisor Dr. Anupam Das
Facebook is one of the biggest social media platforms with 2.6B active users.
It also owns Instagram which has over 800 million active users.
Facebook is also one of the biggest social media advertisement platforms with around
10 million advertisers.
Advertisement is the main source of revenue for Facebook as 98.5% of it’s revenue
comes from running ads alone.
In order to reach relevant audience, Facebook runs targeted ads
These ads can be targeted based on specific user interests that Facebook infer when users
performs different activities on the platform.
Despite of running targeted ads, prior studies have found that Facebook ads can be irrelevant
for the audience.
In this paper, we attempt to demystify Facebook’s interests inference algorithm to find out
why irrelevant interests are inferred.
We answer three research question.
First, we understand what user activities lead to interest inference
Second, we evaluate how accurately these interests are inferred
Lastly, we evaluate how accurate the explanations are that Facebook provides regarding the interests
inference.
We collect data from two sources: We do planned activities on fresh Facebook
accounts and record the interests inferred to evaluate their accuracy.
We also conduct a user study through a Chrome browser extension to evaluate the interests
of actual users.
We recruited 146 participants from Asia, Europe, and US, through Amazon Mechanical Turk.
In order to understand Facebook's interests inference process, we created fresh Facebook
accounts and performed planned activities on them
After the activities were performed, the ad profiles of the accounts got populated with
interests We recorded the snapshot of ad profiles everyday
during the experiment.
To answer research question 1, we performed controlled activities
Our controlled activities were divided into two main types: Positive interaction and Negative
interaction. Positive interaction consisted of Page likes,
liking or love reacting the posts or posting positive comments.
It also included just scrolling through the posts.
The negative interaction consisted of negative activities such as angry reacts or negative
comments on the posts.
One account was solely dedicated for positive interaction and another account was dedicated for negative
interaction throughout the experiment.
Both positive and negative interaction was done on the same set of Facebook pages.
One dedicated account acted as a control on which on activities were performed.
We performed positive interaction on one account and negative interaction on the other.
However, doing only one type of interaction on an account might not represent natural
behavior of a Facebook user because users can perform positive and negative activities
together on their account rather then doing only one type of activity
So, to capture this difference and to see if Facebook infer interests differently when
both positive and negative activities are performed together,
we dedicated one account on which we performed both positive and negative activities on the
same set of pages together.
We also divided the set of pages into two parts, one for each type of activity.
Then we compared the interests inferred by the both type of activities to the respective
interests in positive and negative only profiles.
We found that mixing the activities did not have a significant affect on the inference
algorithm as we saw very similar interests inferred in both cases
Both positive and negative controlled activities were repeated for different locations, gender,
and age groups to capture any differences causes by these factors.
After the interests were inferred, they were mapped to the activities based on timeline
and were labelled as relevant or irrelevant by two researchers.
The researchers performed the labelling independently and the disagreement was quantified using
Cohen’s Kappa metric.
After that, the conflicts were resolved.
We found that interest profiles for both positive and negative accounts were statistically similar
by running Chi Square tests
So, we concluded that Facebook’s inference algorithm did not consider the nature or context
of the activities to infer interests, which can potentially be the reason for irrelevant
interests We also found no statistical difference for
activities across all demographics which indicates that Facebook’s inference algorithm does
not perform differently based on these demographics.
We found that overall 33.22% or around one-third of the interests that were inferred on our
controlled accounts were not accurate.
To evaluate if our findings hold for the actual Facebook users, we conducted a user study.
Our participants installed our browser extension from Chrome webstore which initiated the survey
and data collection.
After obtaining participant's consent, our extension first collected some data from participant’s
accounts.
The data we collected includes top 10 ads with targeting explanations, most recent 100
page likes, number of total activities performed, account age, and the complete ad profile of
the participant.
Our user study was approved by IRB.
Our browser extension generated a dynamic survey for the participants based on their
own data.
We asked that if they are actually interested in some of the interests from their own ad
profile.
We also asked if the interests mentioned in the ad explanations are relavant to them
Finally, we asked if participants see sensitive ads or ads that made them uncomfortable.
We found that 26.47% of the ads that we collected, were targeted based on interests which were
all inaccurate 29.30% of the interests in the ad profile
were irrelevant We also found that 52.7% of the participants
were not even aware of the targeting explanations and 80.8% of them never changed their default
ad settings
We found that participants from India were statistically more satisfied with their experience
with Facebook ads For all other questions we did not find statistical
difference despite the differences Participants in India saw more inappropriate
ads as compared to US and Europe In US, political ads were more common as compared
to other two regions A significant proportion of participants saw
ads related to medication and healthcare in all regions
Now, we evaluate the accuracy and effectiveness of interests explanations to answer our research
question 3.
In the ad profile, along with the list of inferred interest,
Facebook also provides the reason that explains why each interest was inferred,
An example of such explanation is shows in the figure
Through our controlled experiments, We found that the interest explanations
provided by Facebook were very generic and did not
mention the exact activities that cause its inference
From our controlled Facebook profiles, we found that Facebook provided same generic
explanation for all interests that were inferred via different activities.
Finally, based on our findings, we provide recommendations to Facebook in order to improve
interests accuracy and user privacy mechanisms Taking context or sentiment into account may
improve the accuracy of interests. Privacy features such as ad or interest explanations
should be made more accessible and noticeable. The explanations regarding data collection
from user activities should be accurate and fine-grained.
I will end the presentation here and now I will be happy to answers any questions.
Thank you.