Sawasdee ka.
Hello everyone.
This is Mintra Ruensuk and I am pleased to present our CHI'22 paper
"Sad or just jealous?
Using Experience Sampling to Understand and Detect Negative Affective Experiences on Instagram"
This work was conducted at the UNIST Interactions Lab with my advisor Ian Oakley, in collaboration
with KAIST DxD Lab with my co-advisor Hwajung Hong and my colleague Taewan Kim.
Social Networking Services or SNS are everywhere.
3.6 billion people are reported to have at least one SNS account and the mean daily use
time for SNSs is over two hours.
SNS users engage in a wide range of activities such as posting self-produced content or viewing,
browsing and reacting to content posted by their connections.
This prevalent and prolonged media consumption impacts SNS users’ emotional states.
While this is mainly a positive outcome, with many users reporting feelings of relaxation
or calm, it is also widely acknowledged that SNS activity can result in negative emotional
states.
These occur when, for example, users view content and experience negative social comparison,
body comparison or other forms of envy.
While individual experiences of this sort may be unconcerning, the high usage rates
for SNS means they can accumulate over time into more serious problems, such as what is
commonly termed "Facebook depression".
We believe that cataloging these negative experiences is important and contribute two
studies that do this.
Both use Experience Sampling Methods (ESM) to repeatedly survey users about their affective
states over prolonged periods and during the course of their regular day-to-day basis.
In our first study, we seek to complement prior accounts of the emotions experienced
during social media use by capturing a wide range of relevant affective constructs directly
after SNS browsing sessions.
We report on the prevalence of different affective experiences, noting that appearance comparison
and envy show relatively high incidence rates.
In our second study, we focus on these two constructs.
We use similar ESMs but also capture smartphone sensor data in addition to self-reported affect.
We then use the self-report data as labels to train classifiers based on the logged sensor
data, showing the potential to develop affect-aware systems capable of detecting negative emotional
experiences as they occur during daily SNS use.
In our first study, we developed IG Use, an Android app that monitors phone app use and
captures ESM survey data directly after an SNS use session - specifically an Instagram
session.
We recruited 19 participants, all regular users of Instagram, and captured self-reports
over a two-week period, ultimately collecting 837 surveys.
The ESM survey was intentionally broad in scope and included assessments of valence,
arousal, social comparison, appearance comparison, envy, self-esteem, and depression.
To draw conclusions, we first discretized ratings for each construct into binary states
and examined the frequency of each.
The results indicate that the majority of Instagram use sessions led to positive affective
experiences and a minority result in negative affect.
However, negative affect occurred more frequently in passive use scenarios (such as viewing
contents) than in active use scenarios (such as messaging or posting content).
In addition, the negative affective states that were most prevalent, at 16% and 17%,
were appearance comparison, a construct linked to body dissatisfaction, and envy.
In sum, this study extends prior literature on affect and SNS by recording how frequently
different negative affective experiences occur, highlighting constructs that are more prevalent
and, therefore, strong candidates for further study.
Building on these findings, we conducted a second ESM study focusing on passive SNS use
and capturing only self-reports of appearance comparison and envy.
Beyond logging affective states, this study sought also to capture data that could enable
them to be detected or predicted.
To do this, the study used an extended version of the IG Use app that presented Instagram
as a webview, that logged all motion and touch data during passive SNS use sessions and that
recorded surveys immediately after sessions were completed.
We then recruited 22 users and ran the study for two weeks.
We recorded a total of 837 self-reports and associated sensor data logs.
As in the first study, we discretized the self-reports into binary states, We again
recorded predominantly positive affective states: negative affect occurred in 10% of
sessions for appearance comparison and 16% of sessions for envy.
We developed classifiers to predict the self-reported affective states from the recorded sensor
data.
To do this, we split each Instragram use session into non–overlapping 15 second windows and
constructed feature vectors composed of summary statistics (such as mean, min or max) for
each sensor channel.
We then used the binary self-report data as labels to train classifiers.
We explored a wide range of model parameters, including variations in the classification
algorithm, sensor data, normalization procedures and data binning technique for the labels.
To prevent overfitting, we used only default classifier parameters and conducted all analyses
with Leave-One-Out-Cross Validation.
The results indicate that peak performance can be achieved with RBF-kernel SVM, with
motion data alone, with adaptive data normalization procedures based on gradually changing baselines
and with simple data binning techniques based on scale mid-points.
Peak accuracies with these configurations were high: 95.78% for binary appearance comparison,
and 93.95% for binary envy.
In addition, performance for the minority classes of negative appearance comparison
and envy remained high: F1 scores of 78 and 81.
These results suggest it is feasible to detect specific, targeted negative affective experiences
while consuming content on SNS using the sensors integrated into standard smartphones.
A number of limitations impact these conclusions including the small scale of our studies,
their focus on a single image-heavy SNS platform and in our use of a limited number of affective
constructs in our surveys.
It would be interesting to run a study using Ekman’s six basic emotions instead of the
circumplex model used in our first study.
That said, we believe the high performance levels we achieve are sufficient to support
design of diverse affect-aware applications and services on SNS.
For example, we could create a visualization that presents affective reactions, to increase
explicit user awareness of their emotional state, or a filter that allows users to configure
their feed to display fewer contents that evoke negative affective experiences.
We see exploring the design space for affect-aware SNS services as the key next step for this
work.
Thank you for listening.
If you are interested in our work, please feel free to contact us.
Happy CHI days!