Hello. Welcome to my talk. My name is Peng Liao. I am currently a postdoc at Harvard Unversity.
Today I am going to present the project on how we design a reinforcement learning
algorithm for use in a physical activity study called HeartSteps.
I am going to start with a brief introduction. Mobile health intervention becomes a fast-growing
field in the recent years due to the technological advancement. This allows us to not only collect
data in real time but also deliver interventions anytime and anywhere.
The goal of the so-called just-in-time adaptative interventions, or JITAIs for short, is to
use the real-time collected information to inform the delivery of the right intervention
at the right moment to help the users to achieve their health goals and maintain their healthy behavior.
In this work, we develop a reinforcement learning algorithm. So what is reinforcement learning?
Reinforcement Learning is a subarea in machine learning. The goal of reinforcement learning
is to learn how to best select the actions to maximize the total rewards while interacting
with unknown environment.
This work is motivated by our collaboration on HeartSteps. HeartSteps is a physical activity
mobile intervention. There are many intervention components in HeartSteps. Here we focus on
the walking suggestion message. Our goal is to design a reinforcement learning algorithm
that decides at each decision time whether to send the walking suggestion message in
real time to help the individual increase physical activity.
Here is a figure representation for the study design. During each of the 90 days in the
study, there are five user-specified decision times in a day. If the user is currently available
for receiving a message, the RL algorithm uses the current context as well as a summary
of past history to determine the randomization probability for sending the message.
After the five decision times in the day, the algorithm updates the treatment policy using the information
collected throughout the day.
We identify a few key challenges that need to be addressed before applyin RL to optimize
the mobile health interventions. First of all, many online RL algorithms require the
“agent” to interact many times with the environment before starting to perform well.
This is impractical especially in mobile health applications because the users could lose interest and disengage quickly
So the RL algorithm should learn quickly. Secondly, due to the complexity of
the context space and the unobserved aspect of the user's context, the observed
human behavior is often complex to model and exhibits some level of non-stationarity over a
long period of time. As a result, the algorithm should accommodate some model mis-specification
and non-stationarity. The mobile interventions often tend to have
a positive effect on the immediate reward. On the other hand, interventions likely produce
negative impact on the future rewards due to the habituation and burden. So It is critical
for the algorithm to adjust for the longer term effects of the current action.
Lastly, we want to ensure that after the study is over, the data can be used to conduct causal
inference to test and develop new behavioral science theory. We want to ensure that those
analyses are feasible
We develop an reinforcement learning algorithm to address these challenges. At a high level,
there are four main components in the algorithm. The algorithm is built upon a Bayesian low-dimensional
linear model for the reward. And we use an informative prior from a prior study. We develop
a trick in estimating the posterior distribution, called the action-centering, to protect against
the model misspecification and non-stationarity. To adjust for the possibly negative delayed
effect of sending the message, we design a dosage variable by using the discounted sum
of past treatments and use the dosage variable to form a low-variance proxy of the future
rewards. Finally, the actions are selected stochastically akin to Thompson Sampling and
we restrict the minimal probability of selecting each action away from zero. This ensures the
feasibility of causal inference and off-policy learning when the study is over.
The key takeaway for this presentation. In this paper, we developed a RL algorithm for
use in HeartSteps V2. The preliminary validation of the algorithm demonstrates that it outperforms
the standard Thompson Sampling Bandit algorithm in the simulation. We also
assessed the performance of the algorithm using the pilot data from HeartSteps V2 which
shows some initial good signs of the algorithm. The HeartSteps V2 study is now finished. We
are currently working on the data export and planning to further assess the performance
of the algorithm. In this work, the RL algorithm uses the user’s own data to learn the treatment policy
A very important direction for future research is how to properly and efficiently
combine the data from all the participants to further speed up the learning process.
This is the end of my presentation.
Thank you for listening.