Hi everyone , I am mashfiqui rabb. I am a research assistant at Apple.
Today I will be presenting an app called ReVibe. This work is done with my colleague at Harvard and the University of Michigan
Let me start by giving an overview of what is ReVibe
ReVibe is a mobile app to recall earlier moments of the day in the evening.
And ReVibe provides contexts from these earlier moments in the day to make the recall easy.
The idea is rather than interrupting people with Ecological Momentary Assessments or EMAs
and asking how they are feeling for example in the middle of day
ReVibe uses a less interruptive method where we ask people
for example in the midday and we use context such location and activity from midday
to improve the recall of what they were feeling in the midday
Now the question is why would anyone be interested in something like ReVIbe
As you may know, mHealth apps have very poor engagement
Recent market research shows that 75% health app users stop using their app after 10th use
Now, the keyword here is `use`. When you ask people more than use
and interrupt them with EMAs, they would disengage even more
Despite being highly interruptive, EMAs are still widely used and they are still
gold standard for in-the-moment data collection because they can capture subjective data
and they are highly accurate because people reply in the moment as things are happening in their life.
Now given the advantages of EMA people have tried to improve adherence of in-the-moment data collection for a while
the most common approaches are motivating people with money or gamification
but where ReVibe is different is rather than motivating
people, ReVibe uses a different mechanism namely reducing burden
to improve data collection of daily moments
The reason why ReVibe is low-burden is quite obvious
Rather than pining 4-times a day and asking what they are feeling
ReVibe automatically captures context throughout the day
done ping people. And at the end of day when people are less busy
ReVibe asks about moments earlier from the day
and use the contexts it recorded earlier and present them to improve recall
And literature on Human Episodic Memory propose that when people are presented with context
they recall better
but those studies are in controlled settings so we expanded the idea in uncontrolled settings
where we continuously recorded location, physical activity and whether there is human conversation present nearby
and in the evening we asked people to recall earlier moments of the day, for example
moment 11AM earlier in the day, we present location, physical activity and ambient sound
15 minutes prior to 15 minutes after 11AM.
And then we ask the question, what was participants stress, mood, what they were doing, who they were with, and some sort of standard question around mindfulness and sensitive question
Before we go to our user study, I want to mention there is one other low-burden in the literature called micro-EMA.
but micro-EMA asks one question at a time. And their adhrence is quite high.
but scientist are often intested in asking multiple question to capture latent variable or inter-rationshlop between variables
for example, they may interested in whether stress is related to hopefulenss,
So, if scientists want to ask multiple questions, then micro-EMA has to ping multiple times
and if consider what is the adherence of all these micro-EMAs together is way less than 0.9
Moving on to the evaluation study,
so far we have made the assumption that ReVibe has better adherence than EMA, and context can improve recall accuracy
we explicitly check these assumptions in our evaluation study
specifically, our first hypothesis accuracy of recall will be better when context is available
We measure recall accuracy by matching with EMAs at the same moment as commonly done in recall studies
Our second hypothesis is our evening recall has better adherence than EMAs
We ran a micro-randomized trial to test these hyptheses
The study went as follows
at the start of the study participants set up a time they wake up
we denote this time as t_w
in the figure
after t_w, we define four 3-hour adjacent non-overalapping time window
then in each window, we select a time point at random which we call a decision point
then we decide with 0.5 probability whether the ask an EMA or not at a decision point
In the figure we denote a solic circle if we ask an EMA at a decision point and a hollow (white) circle otherwise
At the end of the day, in the evening,
denote by t_r in the slide
we randomly pick 2 of the 4 decision points in the day and ask participant to recall these moments (i.e., time of the decision points)
One of the two (selected) randomized decision points is selected to receive contextual information. We denote the decision point with context with an * in the figure
And this process repeats for 14-days for each participant
14 day is the length of our study and we have in total 54 participants
Moving on to the results of the study
We found our first hypothesis to be statistically signficant. More concretely, recall and EMA answers exactly match
with each 5.6% more when contextual information is available during recall
We also foumd that decision points from afternoon are easier to recall, which makes sesne since afternoon is more recent (thus fresh in memory) since recall happens in the evening
We also found our second hypothesis to be statistically significant as well.
Evening recall adhrence is nearly 27% higher than EMAs.
This effect size is high for behavioral studies
We also found EMA adhrence decrease over study days, but recall adherence doesn't change significantly over study days.
We also ran some exploratory and qualitative analysis
Our first exploratory finding was, raw accuracy of recalls are indeed quite high… for binary question, we found that more than 85% cases
recall and EMA answers exactly matched with each other.
for 5-point likert scale questions, more than 86% cases recall and EMA answers are within one point of each other… Another
Another exploratory find was… EMA adherence is lower for morning moments compared to afternoon… no such time of day difference was observed for recall
Finally, we indeed found that interruption burden of EMA to be higher than evening recall in user burden scale
But, a surprising finding was participant felt more self-reflective during evening recall compared to EMA.
With that I'll end my talk. If you have any question please use my twitter handle to ask me questions. Thank you