Hi I'm Tianshi Li from Carnegie Mellon University.
I'm happy to present our paper "Alert Now or Never: Understanding and Predicting Notification
Preferences of Smartphone Users" on behalf of my co-authors Julia, Miguel, Jason and Jeff.
Notifications are an indispensable feature of today's mobile devices.
However untimely notifications can lead to a variety of negative implications on users.
Researchers have proposed two main mitigation approaches for this problem. The first approach is
to defer the delivery of notifications to the next detected opportune moment. The second approach is
to emulate users actions on notifications, such as discarding notifications frequently dismissed by
users. However these approaches are grounded in assumptions about how users want a system
to deliver their notifications that have not been formally tested. Next, I'm going to introduce key
limitations of these approaches and then introduce our research questions to address these issues.
The first limitation is that deferment-based methods often have the risk of missing urgent
notifications, as it mainly focuses on the context while not take the notification
content into account. The bounded deferral method was proposed as a method to address this issue,
while we still lack a systematic understanding regarding how different factors affect users
notification preferences. Therefore our first research question is what factors
affect users notification preferences and how do these factors affect notification preferences.
The second problem is that users' actions do not always represent how they think.
The authors of PrefMiner have alluded to this issue by discussing an exceptional
case regarding reminder notifications. However we still lack a systematic and rigorous understanding
about the relationship between user actions and their preferences. In our second research
question we want to study what are the reasons that cause users to act upon notifications in a
certain way and how do users' actions on their notifications correlate with their preferences.
The third issue is subjective labels are costly to obtain, which makes it difficult to adapt to
user preferences on the fly. Users' observable actions are easier to obtain at a large scale
while we don't understand much about their relationship with users' preferences.
So in the last research question, we want to build machine learning models to predict user's delivery
preferences solely using user actions and compare it with models built using user labeled data
to tackle the three questions. We conducted an experience sampling study with 35 android users
for a week to directly probe how they wanted their notifications to be delivered and record how they
reacted to the notifications in the meanwhile. Data collection for the study was conducted
remotely via a custom Android app. Over the course of the study participants responded to two types
of ESM tasks triggered by notification events. Task 1 was triggered by notification seen events.
The main question in the Task 1 survey is to select their preferred way to deliver this
notification. The four options were: sent now alert now; send now alert later; send now never
alert; and never send. We also ask the user to provide a free response to explain their choice.
Task 2 was triggered by four types of user actions that remove a notification.
The action type could be automatically detected while we asked the user to specify whether they
took the action immediately after seeing the notification and explain the reason.
Now i'll introduce the key findings we derived from this data set to
answer the three research questions. First we analyzed the 525 Task 1 responses
collected in the first five days. We found that only 56 percent notifications were expected to
be delivered with an immediate alert, indicating a strong need for reducing interruption. However
for other notifications the ratios of "sent now never alert" and "never send" were significantly
higher than "send now alert later", suggesting suppressing the alert was preferred than deferring
the alert. Then two researchers conducted an open coding analysis on users reasons
for their selected notification preferences. This table shows the resulting typology. We found that
91.6% of responses mentioned content-based factors while only 15.3% mentioned contextual factors.
This difference was statistically significant.
Next we analyzed the Task 2 responses and found out that there was no one-on-one mapping between
users' preferences and behaviors. For example sometimes users directly took action via the
notification and sometimes they did not directly act upon notification, such as
opened the app to reply to a message. These two groups represent very different behavior
patterns while the corresponding delivery preferences were very similar. Conversely,
we also noticed notifications that users acted upon in a similar way while the delivery
preferences were very different. In this example users tend to immediately cancel notifications
both if they did not need them or if they had already seen them elsewhere. However users still
wanted immediate alerts for about two-thirds of the notifications that they had seen elsewhere.
Furthermore we found that although some prior work used users' action type or response time alone as
a proxy of users' preferences, our quantitative modeling experiments suggest otherwise.
The accuracy of predicting preferences based on response time alone is only 56.9%
which is equivalent to always predicting "send now alert now". The accuracy based
on action type is only 58.8% which is only slightly better than the chance accuracy.
Lastly we built machine learning models to predict users' notification delivery preferences.
To make the data set more balanced we kept the "send now alert now" option and grouped
the other three options into one new option. This chart summarizes the intuition behind the
three classifiers we built and tested, including a generic classifier, a classifier personalized
on user subjective labels, and a classifier personalized on users objective actions on
notifications. The key idea of the user behavior personalized model is that we first train an SVM
classifier using other users' data to predict preferences using user behaviors. And for the
target user, our algorithm only needs to record how they reacted to previous notifications.
For an incoming notification, our algorithm will aggregate user behaviors on past notifications
with similar content and then use this SVM classifier to generate a preference prediction.
Fortunately our results show that by aggregating past notifications of similar content,
a model of notification delivery preferences personalized on user actions could achieve
similar performance as a model personalized on user labeled preferences. The best performance
of the two personalized models were very close which improved the performance of the
generic model by 42% and 39% respectively. From our findings we envisioned that future
notification managers should use a more nuanced, holistic, and mixed-initiative
approach to ensure both accuracy and user control. Here we illustrate one possible idea
inspired by the "alert now or never" finding and can be supported by our content-based classifiers.
Basically the system will predict an important score for each notification based on its content
and only alert the users about its delivery if the important score exceeds a certain threshold
that can be adjusted by the user. To sum up our mixed-methods ESM
study found that Android users preferred mitigating interruptions by suppressing
alerts over deferring notifications and ascribed their preferences to content related factors
then contextual factors. Although the relationship between your user's preference and behavior
was messy our ml model showed it is feasible to predict notification preferences using aggregated
behavioral data and notification content. Finally we argue that a more nuanced, holistic,
mixed-initiative notification management approach is needed
to both free users from excessive disruption and keep them in control of important notifications.
Thanks for watching and I'm happy to take any questions.