Hello, and thanks for listening to this presentation.
My name is Talie Massachi, and I'll
be talking to you about "Sochiatrist--
Signals of Affecting Messaging Data."
So for a moment, imagine that you are a clinical psychologist
or a therapist.
You're going to see each patient maybe once a week, maybe
once a month, maybe every other week
depending on what they need and what they can afford.
So you, as a therapist, as a psychologist,
are only seeing occasional small windows
into the life and emotional state of your patient.
But now, imagine if you as a therapist or a psychologist
had a way to gain insight on the emotional state of your patient
during that whole period of time since you last saw them.
So in response to this, we present Sochiatrist.
Sochiatrist is a system that extracts, anonymizes,
and then consolidates social media
data for an individual user over a selected time period.
What this means is that a patient
can choose whatever platforms whose data
they want to share with you and then pick a time range of data
that they want to share.
They can anonymize their messages
to keep their contacts' personal information safe,
and then they can present that extracted data to you.
Now, in terms of research, having
a bunch of messages and posts organized into files by study
participant is pretty useful.
It makes it easier to run analyses.
It makes it easier to hand code for specific events,
and it might get you a broader range of data
than you would have otherwise gotten.
And to be honest, there are a few clinical studies
that are currently using Sochiatrist
in exactly this way.
But when we're thinking from the perspective of a clinician,
you don't want to read through pages and pages of social media
data for each patient just to get an idea of what they
were feeling at certain times.
So imagine if you, as a psychologist, as a therapist,
have access to something like this graph where you could,
for example, see that last Tuesday your patient was
feeling really negatively.
Now, you as a clinician can either
ask your patient about that event,
or you can keep it in mind as something that happened
but maybe your patient isn't ready to talk about.
So we focused this study on two automated sentiment analysis
techniques, LIWC and VADER--
LIWC because it's very popular when
it comes to affect detection on text and VADER
because it's a very similar system,
but it's intended to be used specifically
with social media data.
We also focus only on private messaging data
since there are previous studies comparing public social media
data and other textual sources with measures of affect
already.
So in our study, we notified participants three times a day
asking them to fill out the positive and negative affect
schedule, or PANAS as I'll be referring
to it past this point.
And this gave us two scores that we can now
use as a ground truth for our participants' current mood.
So after we collected two weeks of ground-truth data,
we invited participants into the lab
where we extracted and anonymized
through Sochiatrist their social media private messaging
data over that same period.
So for each data point in our final step for analysis,
we selected all sent messages from that participant
for up to 12 hours before and 12 hours
after the point at which they had recorded a given mood
score.
We then tried to detect affect from this data in three ways.
First, we gave the 24-hour window of anonymized messages
to a group of three human coders to write
the positive and negative affect of the participant
as best as they could.
Second and third, we ran each of the automated sentiment
analysis techniques, LIWC and VADER, on that same data.
Each of these values was then compared to the ground truth
PANAS scores that we had collected earlier.
Unsurprisingly, we found that the human estimation
of affect scores was more accurate than either automated
technique.
We did also find though that scores from VADER, that
is, the system optimized for social media data,
correlated much more accurately with ground-truth PANAS scores
than those found by LIWC correlated with the same PANAS
scores.
You'll also notice that VADER scores correlated significantly
with PANAS ground truth while LIWC did not.
Furthermore, we saw that, compared
to previous studies using public social media data,
our results using private messaging data
correlated much more closely to ground-truth PANAS scores.
You'll also notice that our results using private messaging
data are comparable to previous studies using
LIWC on other sources of text, such as diary
entries or natural speech.
And you can see further discussion
on why this might be in our paper.
Finally, through qualitative analysis,
we identified three pitfalls that
seem to lead to worse predictions of affect
that we called "mispredictions."
The first of these was overreliance on tone
or context, the second of which was message weight based
on temporal distance where temporal distance is defined
as "the distance in time between a message
and when the PANAS score for that session was recorded,
and, finally, the inability to dissociate
positive and negative affect from each other.
So in total today, I addressed several points.
First, I introduced Sochiatrist, a system for social media data
extraction, anonymization, and compilation.
Second, we ran a study investigating
affect detection using automated sentiment analysis
systems, LIWC and VADER, on extracted private messaging
data.
We found promising results when compared with analysis
on public social media data.
And we pointed out several areas of potential improvement
for these automated systems.
In conclusion, private messaging is a promising data source
when it comes to affect detection,
and further investigation seems warranted.
And finally, finally, thank you so much for listening
to this presentation.
If you want more information on Sochiatrist,
you can check our website, or you can check out
the paper "Sochiatrist--
Signals of Affect in Messaging Data."
Thanks so much.
Bye.