I am Geza, and this paper is about adapting to user difficulty preferences
in the context of behavior change systems.
Attrition is a major problem faced by behavior change systems.
In prior work where we investigated causes of attrition, we observed
that some users uninstall because they think interventions are too aggressive.
At the same time,
other users uninstall for the opposite reason — they think interventions are too easy.
Since these users have conflicting difficulty preferences, we cannot
simultaneously satisfy both groups unless we adapt to users.
How should behavior change systems adapt to difficulty preferences?
We investigate this problem using HabitLab,
a browser extension we developed that helps users reduce their time online
After installing HabitLab, users select sites they would like to reduce time on,
such as Facebook, which we call goal sites.
Each time a user visits a goal site, an intervention will be shown to the user.
This paper focuses on users whose goal is to reduce their time on Facebook.
HabitLab has 12 interventions available for reducing time on Facebook.
We asked raters to classify each of HabitLab’s
12 Facebook interventions into one of 3 difficulty levels: Easy, Medium, or Hard.
Here is an example of an Easy intervention.
It shows a timer in the user’s Facebook
feed displaying how long they have spent on Facebook today.
Here is an example of a Medium intervention.
It hides the Facebook feed by default.
Here is an example of a Hard intervention.
It automatically closes the tab after 60 seconds.
We use HabitLab to investigate the following research questions.
First, we need to determine whether users’ difficulty preferences
change over time - if they do, this means systems need to periodically ask users
to check whether their preferences have changed.
To determine how difficulty preferences change over time, we need to know what users’
preferences were during the onboarding process, when they installed HabitLab.
To determine users’ initial difficulty preferences, we show this prompt to new users,
asking them to choose one of four difficulty options - no interventions, easy, medium, or hard.
We have responses from over 8 thousand users.
The most commonly chosen initial difficulty is Easy,
followed by Medium and Hard. Less than 10% of users choose “No Intervention”.
To measure how difficulty preferences change over time, each time a user visits
Facebook, we prompt them to choose the intervention difficulty for this visit.
Here we show how difficulty choices change over the first 200 choices.
We see that the fraction of users choosing “No Intervention” increases over time,
with over 70% of users choosing “No Intervention” on their 200th choice.
Thus, users’ difficulty choices indeed change — they increasingly
choose “no intervention” over time.
Since users’ preferences change over time,
we need to periodically prompt users to detect these changes.
We hypothesize that excessive prompting could annoy users and lead them to uninstall,
so we study whether the frequency of difficulty prompts affects attrition.
We randomly assign new users into 4 conditions, which differ in how frequently the user is shown
the difficulty prompt: on 0% of visits, on 25% of visits, on 50% of visits, or on 100% of visits.
We tracked over 1000 users over 528 days,
and compared attrition via a Cox Hazard Regression model.
Attrition is lowest in the condition where users are shown the difficulty prompt on 25% of visits.
Attrition in this condition is significantly lower than asking users to
answer the difficulty choice prompt every visit.
Thus, we found that while prompting users to answer the difficulty choice prompt
every visit increases attrition, low-frequency prompting does not increase attrition.
Finally, we wanted to see what would happen if
we put users in control of the difficulty prompting process.
If given a choice, how often do users prefer to be asked about their difficulty preferences?
To gather this data, after users answer the difficulty prompt,
we also ask how long to wait until prompting them again, shown on the right.
We find that users choose “No intervention this visit,
but ask again next visit” over 70% of the time.
This is surprising - users are asking for frequent prompting,
despite their difficulty choices remaining the same.
Perhaps while answering the time prompt, users were expecting that their future difficulty
choice might change, but when the time actually comes, they still keep choosing “no intervention”.
Thus, if we let users choose when they will be asked about their difficulty preferences,
users’ most common choice is to ask for no intervention now,
but request that they be asked again next visit.
In summary, users’ intervention difficulty preferences change over time,
and by using low-frequency prompts we can measure the change with no costs to attrition.
However, given the choice, many users end up choosing no intervention this
visit, but want to be asked again next visit.