Hello! My name is Varun Mishra, and today I will be presenting our work on reproducibility in
physiological stress detection. This work was done in collaboration with colleagues
at Dartmouth, Northwestern, Washington university in St. Louis, and IBM Research
So what is stress?
Simply put, stress is the brain’s response to a challenging or demanding external stimuli.
Let’s do a quick review of the Autonomic nervous system.
When an individual is experiencing stress, the sympathetic nervous system contributes
to what is called a fight or flight response. It signals the adrenal glands to release adrenaline
and crotisol and results in a physiological impact like change in heart-rate and breathing,
and further increases sweat gland activity.
On the other hand, once the person stops experiencing stress,
the parasympathetic nervous system engages to help recover and reverses the effect of SNS.
So occasional stress is actually good!
In fact, most of you in the audience must have experienced that feeling stressed a day before
a paper deadline pushes us to work harder! But continuously feeling such incidents of acute
stress leads to a “wear-and-tear” on the body, and could eventually lead to negative health outcomes.
Due to the advances in sensing technologies,
researchers have used sensors to measure physiological changes to detect stress.
In fact, one of the first works on using physiological signals to detect stress
was done back in 2005… and people have been studying it since.
So we are still trying to detect stress after 15 years!
There are several challenges that exist. One of the major challenges that our work focuses on
is the lack of reproducibility and large-scale deployments.
This is because of reliance on custom-made or expensive devices and sensors.
Researchers often build their own custom hardware or use expensive clinical-quality hardware which
makes it inaccessible conducting large deployments. Further, in most of the prior
works, the researchers build stress-detection models using data collected from participants
belonging to a particular demographic, in a specific setting, using a particular device.
Such an approach makes it difficult to evaluate the efficacy of the models
applied to a broader population, or to compare with the methods used in other studies.
The model reproducibility is either assumed or left as future work,
So we collected data from four different studies. The studies had different study protocols. Two
studies used the polar heart-rate monitor, and the other two studies used the Empatica E4.
Across these four studies, we have introduced some degree of heterogeneity
with the different study protocols, populations, and devices.
For more details about the individual studies and the study protocols, please refer to our paper.
When we started building stress detection models, we observed that the stress detection methods in
prior works consider each time-window that is used for classification as independent events.
In doing so, they do not account for prior stress and miss out on the temporal nature of stress.
In our paper, we discuss different methods to account for prior stress and found
that our methods consistently performed better than the approach in prior works,
across all datasets, i.e., our method was reproducible across studies.
Due to time limitations, I will not be discussing the exact results from our method,
so if you are interested, we urge you to read the paper. In this talk, I will focus on our
evaluations when we apply models trained on data from one study to detect stress
in another study, i.e., the reproducibility of the models.
Now, we move forward and test the reproducibility of individual models. So now we will evaluate
the performance of model trained using one dataset
and tested on a different unseen dataset. The models output a probability of stress.
We used a threshold based on the training data to classify as stress or not stressed.
In the actual paper, we present the full comparison of each model with every dataset,
but for this talk, let’s visualize the performance of the models when evaluated with the S2 dataset.
So models built using the S1, S3, S4 datasets were evaluated with the S2
dataset. For comparison, I’ve included the cross validation results of the S2 model.
So we see that the F1-score from the cross-validation of S2 performs best.
However, looking closely, we see that for models trained using a different dataset,
there is a large variation between the precision and recall. This suggests that the model might
be working well, but it’s the decision making process, i.e., the eventual classification,
that seems too selective. So to test this hypothesis, we decided to customize the
classification threshold for each model based on what worked best for the testing dataset.
We immediately observed that the F1 scores were much better and consistent across all the models.
But on plotting the thresholds, we see that each model had a separate threshold that
was used for testing on this particular test set, i.e., S2. This threshold, however varies
widely between model to model and seems to be dependent on both the training and test data.
Hence at this point, it seems that models trained on one dataset might not be enough
or even very accurate to detect stress with participants from a different study!
To get a better sense of what might be happening, let us look at the stress
signal dynamics for a random participant from S2.
So, we plot the likelihood of stress from each model for this participant;
the dotted lines represent the optimized threshold for classification, and the shaded regions mark
the baseline and different stress-inducing tasks.
are similar but the actual values differ quite a bit based on the training data.
So this further strengthens our assumption that just choosing a fixed threshold based
on the training set might not work! But we cannot do away completely with
a threshold either. If we want to provide an intervention then some quantitative threshold
is needed to determine when to deliver the intervention and to observe its effectiveness.
So ideally, we would want to determine a custom threshold without any knowledge of
the data in the test sets. One thing we know is that the likelihood ranges from 0 to 1. If the
values are closer to 0 then the person is less likely to be stressed and if it is closer to 1,
then the person is more likely to be stressed.
So we tried out a clustering-based approach. We set two clusters with centers at 0 and 1
and fit the probabilities for each participant as we had new likelihoods were calculated.
To visualize what’s happening, we plot the results for S1 and S4 signal for the same participant.
We observe that the assignment of labels of stressed and not-stressed
was quite consistent across this user. So we potentially have a way to determine a
threshold of stress in an unsupervised way! We conducted thorough evaluations of our
approach across all combinations of train and test datasets.
We found that our unsupervised approach consistently consistently performs better than
choosing a fixed threshold from the training set. In fact, in some cases,
it even outperforms using an optimized threshold from the test set! This is great because now these
models can actually be deployed in real-world where there can be no “optimized threshold”.
Your evaluations show that our methods and models are indeed reproducible.
As next steps, we are currently evaluating our models on additional datasets to test
for potential generalizability. We are also planning to deploy these
models in free-living conditions and evaluate their effectiveness.
Thank you for listening, and please feel free to reach out to us for more details.