Hi, I'm Christopher Collins
Today I'll be presenting Covid Connect
our chat-driven anonymous story-sharing system for peer support
A collaboration between Ontario Tech University, Ontario Shores Centre for Mental Health Sciences, and the city of Oshawa
I'm joined today by my co-authors: Nathan and Zixin
to help with the presentation on behalf of our other collaborators Simone, Shawn, and Jennifer.
In addition to illness and death,
Covid-19 brought immediate and drastic changes to our lives
from empty classrooms, closed playgrounds, and product hoarding
to days of boredom, warnings, and Zoom birthdays
The direct and indirect effects of the pandemic were numerous
It’s within this context we decided to create Covid Connect
In fact, some of the pictures you see here are from our own lives
So as we were also living through the pandemic
we were inspired to take an autobiographical design approach
acting as both designers and the representatives of the target audience
Our aim was to create something quickly, and continuously improve it through iterative development in order to address the pressing need
At the height of the pandemic, accessing mental health support was limited
Moreover, many people could not afford, did not have the time,
or were not comfortable talking to a person about their problems
So we were inspired by other similar low cost and automated approaches to providing support
With a therapist chatbot like ELIZA, created by Joseph Weizenbaum in the 1960s
saw that simple chat scripts were sufficient to engage and encourage people to share personal stories
Togetherall was another source of inspiration for our project
they provide a combination of one-on-one help and peer mental support help
Another source of inspiration is a community art initiative called PostSecret
They invited participants to mail postcards to curators of the site where they would be displayed
Because people were allowed to be anonymous, they shared personal and thought-provoking stories
We wanted our system to be free and accessible across multiple platform
and since mental health is a private subject
we anonymized the messages and allowed users to request to delete their messages from our database
Due to the gravity of the pandemic
we deployed our system as quickly as we could and used iterative refinement
To understand what our target stakeholders' concerns were
we conducted a survey and gathered a focus group of HR professionals at our local municipal government
The survey and focus group revealed that most common themes reported by participants
were related to school, working from home, loneliness, and more
With these themes,
we hoped to create an inviting and engaging chat script informed by professionals within the mental health sector
We were advices to use open-ended questions, acknowledge and affirm,
demonstrate patience, end on a positive note, and invite further participation
The chat agent attempts to detect the intent of the user's messages
and generate appropriate responses to 3 sentiment categories and 10 of the previously mentioned topics
When the agent can't detect the user's intent is
it chooses one of nine redirect responses provided our mental health professional team members
for example,
“What is the best thing that happened to you over the pandemic?”
We underwent several rounds of sketch-based brainstorming with our full lab to refine the user interface and layout
with the aim of integrating the chat area with some type of visualization of previous statements
The final design centred around the idea of bubbles, containing the full test of previously submitted stories,
and a central chat window for dialogue with the agent
Due to the stressful nature of the context in which the project was deployed
We decided to use a more cheerful colour scheme
We based the design on a metaphor of bubbles, which are temporary, colourful, and move in unique patterns
The final design removed the border between chat and bubble to integrate it as a single application
The chat window relates to a zoomable canvas which is continuously populated with bubbles
which show previously submitted stories related to the current chat
We intentionally do not use summarization or aggregation visualizations,
instead showing full texts to honour the individual stories submitted
Numerous features were implemented In the interest of maintaining user privacy,
such as toxic comment screening,
detecting distress in the chat and redirected to emergency services
detecting and anonymizing named entities
and allowing users to delete their data at anytime with a unique code
In addition, there is a three day data holdback for user data
The application starts with a welcome screen that describes the purpose of Covid Connect
When the user decides they want to chat
they are taken to a consent form
and they must scroll to the bottom and pass a CAPTCHA to proceed
Once inside the application, the participant may begin to send messages immediately
After the participant sends their first message,
we use natural language processing techniques to search the database for similar texts
Then render those texts as bubbles on a zoomable canvas in the background
The bubbles can be selected to zoom in and provide "up votes" or feedback
At the end of the chat session, the agent provides a link to a feedback survey
CovidConnect is free to use and was created using open source libraries
All data is stored on our private servers and the chat AI is locally hosted
Due to the urgent nature of this research,
we had a fast launch and use iterative refinement through observing logs to fix any bugs and improve the system
Here we have the complete system diagram which comprises of three logical groupings:
Frontend, Web & Data Backend, and natural language processing which each have discrete components
Communication between components is denoted by edges
which are also colour coded to denote the class of data being communicated.
The frontend consists of the chat bubble visualization, the chat box and user and data management functions
The chatbox communicates directly with the Rasa Agent to enable a rich conversation experience
In turn, the agent triggers chat bubble visualization updates
ensuring the contents of the most recent bubbles is similar to the topic the user is currently discussing
Natural Language processing is primarily composed of customized Rasa Open Source components
The Rasa Agent implements the NLU pipeline and dialogue policies
At the highest level
chat messages from the user enter the NLP system
and it in turn replies with the agent response to the chatbox
and sends event messages with parsed data to the message broker
which are acted upon by the Web & Data backend
The Web & Data backend is responsible for the system logic and control
It ties together the disparate components via the message broker,
interacts with the database,
scrubs message of personally identifying Information and archives them,
performing message similarity searches
and also implements user data controls.
We launched Covid Connect in March 2021
and while it remains online
our data snapshot was taken in February 2022
Recruitment was carried out by direct emails to employees and student at partner institutions
We also posted on social media posts, and used posters
This project was also featured in a news story and
additionally, we invited people to forward the recruitment to their networks
For our data collection, we had a total of of 409 unique sender IDs
There were 1700 user messages
Most of the messages we obtained consisted of greetings and simple “yes/no” responses
So to conduct qualitative analysis using open coding
we filtered the messages and only analyzed those with more than 5 words
The analysis was conducted by a two pass method
involving a researcher going over all the entries manually
and then consulting a mental health expert co-author to discuss data groupings and overarching themes
Since our system was created to act as a story-sharing platform,
we found that many of the messages started with the pronoun “I”
To understand any patterns,
we created word trees of messages started with “I”
Here you see those that start with the phrase “I miss”
showing that people miss going to places and seeing their families, friends, and loved ones
Similarly, looking at the word tree created using messages starting with “I have”
we can see that people wanted to share their thoughts with our agent
Telling them that they weren’t feeling the best or had nobody to talk to
They also told it very general daily status reports like
“I have a runny nose”
or “I have an appointment today”
From these messages, we can see that participants were able to have meaningful conversations with the agent
despite the simple chat script
Through axial coding, we detected 11 high level categories
with some messages belonging to two categories
The higher level categories were further clustered into three themes: emotion, experiences, and bonding
The larger categories of the emotion based messages were derived using Dr. Gloria Wilcox's feeling or emotion wheel
The main emotions expressed in the data we collected were
hopeful, sad, angry and bad
Some expressed frustration with their life, work, restrictions, and so on
The other theme that was expressed in our data were people talking about experiences or topics
For example the melding of personal life with work life affected people's relationship with those they live with
We found that people generally interacted with the agent as if it was a confidant or friend
There were messages asking for help, expressing personal emotions, or having a small talk
Overall, the types of information people shared with Covid Connect
were surprisingly heartfelt, emotional, and relatable to our own experiences during the pandemic
An alternative framing of our analysis
was to examine the data from the perspective of challenges and coping mechanisms
Many poignant stories of pandemic challenges were submitted
for example
“Because I’m far from home
I feel like everyone back there is getting on with their lives and I’m just stuck waiting for life to start”
The challenges were met by creative and diverse coping strategies
for example one user wrote
“I’ve connected with my neighbours a bit more
because we are all at home so much, we see each other around a lot"
In addition to analyzing the data collected
we also reflected on the design process and the outcomes of the deployment
We found that the simple chat script was effective in engaging people
even though the agent had a relatively limited vocabulary of statements
The use of open-ended statements was surprisingly effective at eliciting detailed and moving personal stories
sometimes from ourselves as participants
However, we did notice that chat logic failures may have discouraged continued use of the system,
as people provided chat feedback directly to the agent
As the use of an external feedback survey was not well used,
in future we recommend collecting feedback directly through the chat system itself
We do see opportunities for the use of such a tool in additional scenarios
for example anonymous peer support story sharing systems for addiction recovery
We have presented Covid Connect
which remains online and available for you to use
To share your pandemic stories and read the experiences of others
at covidconnect.me