Unknown: Hi, my name is Rachel Kornfield. And I'll be
presenting our paper, "Involving crowdworkers with lived
experience in content development for push based
digital mental health tools: Lessons learned from
crowdsourcing mental health messages."
The problem motivating our work is that our mental health care
system is broken. Mental Health problems are extremely common.
In the US, 20% of Americans each year have a diagnosable mental
health condition. But most of those individuals do not receive
any care at all. Digital mental health tools, like apps, online
programs, and automated text messaging programs have
potential to dramatically expand the convenience and access to
support but they have not been very engaging to users in real
world settings. We wanted to understand whether peer
production methods, specifically crowdsourcing, can help to keep
content engaging within these tools. Those who have firsthand
lived experience may better understand end-users' day-to-day
needs and be able to craft messages to better meet these
needs, in communication styles that those end users will be
receptive to. Crowdsourcing methods also allow for
scalability, and can produce enough content to sustain longer
term engagement with digital mental health tools. Many
individuals with mental health conditions are not interested in
directly communicating with others, but they may be
receptive to peer support that's delivered through automated
tools. So our focus here is on push-based automated messaging
programs that can integrate crowdsourced content. And these
sorts of programs can reach users in their day to day lives
at different times of day with text messages or notifications,
so they can overcome issues with motivation or forgetting to
engage with a tool, potentially keeping users engaged for
longer.
It's important though, to understand what sorts of
messages these end users might be receptive to, and what sorts
of topics of messages crowdworkers would be especially
well-suited to produce.
So I'm going to focus on the first two research questions
here. The others are covered in the paper. First, we wanted to
understand the types of mental health messages that crowd
workers most commonly would compose for push-based automated
messaging tools. Second, we were interested in how crowdworkers
would evaluate messages that were composed by their peers
versus those composed by experts.
Participants were recruited through Mechanical Turk and they
completed the study activities on Qualtrics. All participants
self identified as having experienced a mental health
concern in the past year. In the first wave, 516 Crowd workers
contributed to content development. First, we had them
review expert-composed messages from an existing messaging
program, and they gave feedback on those messages as well as
editing them. We also asked crowdworkers to propose their
own message topics and to compose example messages. In the
second wave, 309 Crowd workers evaluated those same
expert-produced messages, as well as crowdworker-produced
messages from wave one and spoke more broadly to their views of
automated messaging programs.
At the end of wave one, we asked crowdworkers about possible
message topics or messages that we should include that hadn't
been emphasized by experts. So, the prompt read "In the box
below, please write a message on any topic at all that you might
like to receive to help you manage stress, anxiety, or
depression, or tell us what kinds of messages we should
focus on creating." So some responses provided example
messages.
There were 206 example messages produced, and then other crowd
workers gave examples of topics they were interested in. And
there were 249 topic suggestions.
So we grouped these topic suggestions together
into categories. There were about 20 categories, and then we
assigned the example messages to each topic.
The table here shows the most popular topics as far as the
messages produced, and the six most popular were novel topics
introduced by the crowdworkers, meaning that they had not been
in the original expert message set. The first three,
reassurance, affirmation and inspiration, could be considered
emotional support messages. They focus on meeting the emotional
needs of end users. The next three, relaxation, self care and
positive thinking, might be considered self help strategies.
Those bottom five were in the original message set, but they
were endorsed by crowdworkers or those crowdworkers produced
messages that aligned with those categories.
In Wave Two, we focused on evaluating messages in those 11
most popular topics, so we asked crowd workers to rate a set of
messages reflecting those 11 topics and content composed both
by crowd workers and by experts. They were asked, "Imagine you
signed up for an automated text messaging program to help with
mood and mental health. How interested would you be in
receiving each of the following text messages?", rating them on
a Likert scale from one to seven.
We found that there was no significant difference in the
ratings given for expert-composed versus
crowdworker-composed messages.
There was a trend favoring experts. We also looked at
whether there was an interaction between the topic of messages
and the author type.
So what we found here was that experts did have higher ratings
in one category, which was behavioral activation, a
significant interaction effect. And we looked to some of the
specific messages produced by experts in crowd workers to
understand what might be going on here. We did notice that
crowdworkers tended to promote very specific immediate
behaviors like "go outside" or "listen to happy music," whereas
experts tended to prompt the reader to identify activities
that might be personally rewarding.
I just want to highlight some conclusions and future
directions of this work. First, our data support the idea that
crowdsourcing is a viable avenue to rapidly develop content that
might be accessible, and well received by end users. And we
found that they tended to emphasize self-help and
emotional support topics. However, expertise may still be
important for certain topics such as positive activities,
which may relate to really understanding the therapeutic
model behind it. In this case, behavioral activation comes from
CBT and relates to the positive activities topic, so experts may
have benefited from some training in that area. We think
that future work should examine the attribution of content: How
would messages be received if they were specifically
attributed to peers versus experts? And how should that
happen within a messaging program? We may also look at
whether crowdsourcing affects crowdworkers in positive ways,
whether they enjoy and benefit from actually producing content
from others, in which case prompts to write content might
be built into the interventions themselves. There's an example
of that shown on the right hand of the slide.
We may also want to understand how content is received, not in
a study context, but in the context of users' real lives,
how do they receive messages from their peers, and really
understand the effects of interventions that include
crowdsourced content.
So I will end there. Thank you very much for your attention.
And please feel free to contact me with any questions about this
work.