Hello! My name is Varun Mishra, and today I will be presenting our work on detecting
receptivity for mHealth interventions. This work was done in collaboration with colleagues
at Dartmouth, ETH Zurich, University of St. Gallen, and the National University of Singapore.
Recently, there has been a lot of focus and interest in leveraging ubiquitous technologies
to deliver health and well-being interventions. One intervention framework that is particularly
popular is Just-in-time adaptive interventions. JITAIs have been deployed in several studies,
and have shown to be effective for a variety of health outcomes. Fundamentally, JITAI aims
to deliver the right type or amount of support, at the right time.
So what is a “right time?” There are two key concepts when it comes to timing. First
is a state-of-vulnerability, i.e., period of heightened susceptibility to a negative
health outcomes. For example, if the goal is to provide stress-relieving interventions,
then, ideally, the interventions should be delivered when the user is stressed or is
about to get stressed. Here, stress is the vulnerable condition. Delivering stress-relieving
interventions at a time when the user is relaxed might not be beneficial.
The next concept is state-of-receptivity, i.e., periods when users are able and willing
to receive, process, and utilize the support provided. For example, Delivering a stress-relieving
intervention when a person is giving a talk might not be a good time, even if the person
is stressed.
Hence, detecting -- both -- vulnerability and receptivity is important for an intervention’s
effectiveness. In our work, we focus on the receptivity component.
In our prior work, we conducted a 6-week long study with 189 participants in switzerland,
where we deployed a chatbot-based digital coach -- called Ally -- that provided physical
activity interventions aimed to improve daily step goals.
Through that initial study, we defined the various metrics of receptivity that are also
utilized in our current work.
Ally initiated each conversation with a “greeting” message, like Good Morning, or How is it going?
We defined several metrics to measure “receptivity” around these initiating messages.
First, is just-in-time response, i.e., did the participant reply to Ally within a short
period of time. In our analysis, we used a window of 10 minutes.
Next, is response delay, i.e., the time taken to respond to the initiating message.
Third, is conversation engagement, i.e., did the participant engage in a conversation with
Ally.
From the Ally Study, we reported several interesting associations and findings.
One of the important findings was the feasibility of building machine learning models that could
leverage the various factors and detect receptive moments.
In this work, we decided to build on our prior work by actually deploying the machine learning
models with participants in the real-world.
In the initial Ally app, the server sent a push message at random times during the morning,
afternoon and, evening blocks, and the app displayed the message and notification as
soon as it received the push.
In this work, we modified how the app notifications worked. The server still sent a message at
the random time in the morning, afternoon, and evening, but the message was “silent”
and was intercepted by the “receptivity module” that we added to the app. The receptivity
module would use the contextual information available at that time to determine if “now”
was a good time to deliver the message. If it was then the app would notify the user
else it would try again after a few minutes.
We implemented three different intervention delivery methods in our app, and the receptivity
module would choose a method randomly when it receives a silent push from the server.
The first once is A Static Model: which was a random forest
model that remained the same for every participant for the duration of the study.
Next is an Adaptive model: which was a logistic regression model that was re-trained on-the-fly
using data from individual participants And a Control Model: that simply delivered
the prompts at random times.
We trained the ML models using the data collected from the 141 iOS users our previous Ally study.
The Ally+ app was only available on iOS. For more details on how the models were trained
please refer to our paper.
Here’s a quick schematic diagram of the entire process.
At a random time in a time block the server would send a silent push to the Ally+ app.
The receptivity module would receive the push and randomly decide which delivery model to
use. For the first 7 days it would only choose between the static or the control methods.
After the first 7 days, once sufficient user-specific data was available, the receptivity module
would randomly choose between static, adaptive or control methods. If the control method
is selected then the notification is triggered immediately. However, if an ML method is selected
then the app uses the in-the-moment contextual information to determine if the user is receptive.
If model predicts that the user is receptive then the notification is triggered. If not,
then the model kept trying every 5 minutes for 30 minutes. After 30 minutes the app just
triggered the notification as control.
After the notification was triggered, based on how the user interacted with the notification,
the adaptive model was updated for future predictions.
So to test the efficacy of these models in the real-world, we conducted a 3-week long
study with 83 participants from the US East Coast.
Our analyses revealed that over the course of the study participants were upto 40% more
receptive to messages delivered by the static model as compared to the control method.
The messages delivered by the adaptive model did not show any significant difference in
receptivity than the control model.
However, when we looked at the day-by-day trends, we noticed that over time, the adaptive
model’s performance improved. Consider this plot of likelihood of conversation engagement
and the day in the study. We notice that the static model led to consistent engagement
levels over the study duration. The control model, i.e., delivering prompts at random
times led to significant decline in the likelihood of conversation engagement. However, the adaptive
model led to a significant improvement in conversation engagement over time.
In our paper we discuss the various analysis methodologies, results across the different
dimensions of receptivity, and the limitations of our work. We encourage you to read the
paper for more details.
Our results are very promising, and show that it is indeed possible to detect receptive
moments for intervention delivery.
As next steps, we are currently building better models that move beyond just momentary features
and consider more temporal information.
We are also planning another deployment with a longer study duration to evaluate how different
personalized methods perform.
Thank you for listening, and please feel free to reach out to us for more details.