Hello. My name is Estelle Smith, and on behalf of my entire authorship team from across CU Boulder,
the University of Chicago, and UNC Charlotte,
I am really excited to be presenting our work today titled, "The Impact of Governance Bots on
Sense of Virtual Community: Development and Validation of the GOV-BOTs Scale."
Researchers from community and organizational psychology have been studying communities,
what defines a community, and how people experience a sense of community
for many decades. However in the last 20 years, we've seen that communities of humans have been
going online to form online communities. This work focuses on Reddit, a platform that hosts
tons and tons of communities, interconnected in a kind of an ecological system
of communities online. One thing that Reddit has been, we've been seeing increasing numbers of bots
showing up in Reddit communities. So we call these bots "governance bots,"
and these are having profound impacts on communities, because historically we've
always thought of communities as being composed of humans. But now that bots are in these spaces,
acting as social actors, this could have some really important implications on our experiences.
So, I'll start off by asking, what are governance bots? The most common application that people
immediately think of is moderation. So bots such as u/Automoderator which can do things like ban
users or delete comments that that are not allowed in the community. There are really beneficial
impacts of moderation bots, such as norming the community or helping people to feel that they're
safe there, and having the types of content that they want to see be present in the community.
However, they can also potentially raise tensions when users believe that the moderators are using
bots in a way to manipulate them, or when they are startled because it was a bot rather than a human
that that responded to a comment, and these types of tensions potentially could have
negative impacts on users. But these aren't the only types of bots that we see, so for example,
there are some bots that do things when you summon them, such as converting distances--something
that's not super super easy for humans to do in our heads but for bots it's easy computationally.
They can also monitor communities and make posts or comments in response to activities that they
see happening online in the platform. Finally they can just do things for fun. For example,
if they notice the word "cat" showing up in a in a subreddit, maybe a bot shows up and just
gives you a funny picture of a cat. So, all of these different functions of bots can have
really community-gelling effects. They can really improve people's sense of community potentially,
or they might have some negative tensions, and that's really what our study is trying to get at.
We know that there's philosophical and ethical considerations that need to go into bot design,
but our study is primarily focused on the psychological implications of bots in online
communities, and we use these three research questions to guide our study. The first is,
(1) how can we measure users cognitive assessments of bots so whether users perceive those good or
bad impacts on the community? (2) How do bots impact user's sense of virtual community? (3) And
how do users experiences of community vary across Reddit? You can turn to the paper for
more on that third research question, but for the sake of time, I'll skip it in this presentation.
Regarding the first research question, measuring users' cognitive assessments,
the gold standard for understanding users' internal psychological states is to use
psychometric scales. These are those scales where we see a statement with a user's ability to rate
that statement according to how much they agree with it or how much it applies to a to a given
context or a situation, and we used established methods from organizational and community
psychology to develop our scale. First, we did item generation and selection based on prior
literature on bots. We deployed our survey across 12 subreddits, and we use Confirmatory Factor
Analysis (CFA) to ensure that the constructs that we're measuring are actually relating back
to the concept that we're trying to measure inside of a user's cognitive state. Finally,
Structural Equation Modeling (SEM) helps us to understand how the constructs that we measure are
related to each other, what types of pathways of influence do these constructs have on each other.
After performing Confirmatory Factor Analysis, these are the items that survived
in our GOV-BOTs scale. This is the final scale. We came up with four items that measure
positive aspects of Bot Governance, such as providing positive resources,
generating positive interactions, helping the sub to be safer, and removing content. We also
generated another construct called Bot Tensions that measures users' perceptions of when they
think bots are manipulating the sub, upsetting users, or causing causing startling issues in the
subreddit. So this is the full GOV-BOTs scale, and we encourage you to use it in your studies.
For RQ2, understanding the impact of bots on sense of virtual community, we adopt an emergent model
from organizational psychology that looks at antecedents as precedents, things that people
experience before they develop a sense of virtual community (SOVC), as SOVC as its own construct,
and outcomes of SOVC that might result when we have people experiencing a sense of community.
We chose to look at similarity and interactivity, because those are the two most commonly studied
antecedents, and we know, because it's been replicated so many times, that these two factors
can contribute to a sense of virtual community, and we added Bot Governance and Tensions.
There are other options for antecedents that we could have chosen, but the two that we picked are
the most well studied and best best understood, so we chose a minimal subset of constructs,
so that we didn't make our survey too long. We used the these three items to measure sense
of virtual community. Those are coming from prior literature. And we measured one outcome,
identification with the group, which again is one of the most well studied concepts that comes out
as an outcome of sense of virtual community. This is the results of our Structural Equation Model.
importantly, we saw that we replicated well-known relationships. Similarity and interactivity both
positively contribute to SOVC, and SOVC positively contributes to identification with the community.
This gives us a lot of confidence that our survey data was good, and that the methods
that we're using are sound, because we're able to confidently replicate known theory.
We also saw, importantly, that Bot Governance does function as an antecedent to sense of
virtual community, and that it's a positive relationship. So, how to interpret this is
that when users perceive bots having positive governance effects on their community, they
are more likely to experience a sense of virtual community in this space. This is a psychological
conclusion to draw based on a well-validated tool. We didn't see the same effects with Bot
Tensions--that's a little bit more complex, and you can read about that in the paper.
There are some methodological implications here from this work. The first is that HCI
hasn't always done a great job of adopting standards from organizational psychology,
and the concept of construct validity is really important. If we want to make sure that we're
studying what we think we're studying, then we need to use these types of methods in order
to improve the rigor of our work. We believe that this type of interdisciplinary oversight
offers a great solution to construct validity. So that's one implication,
but the others are the pragmatic applications of how we can actually use this scale that
we developed. We believe that it empowers community-centered design of governance bots.
So moderators, for example, can use this tool to track how people are feeling about bots in
the community, and maybe make adjustments if things aren't going so well. It's also just
good to know over time how people are perceiving bots in the community, and this tool will enable
that. It's also useful as a research tool for empirical studies, so if there's any time we want
to understand how people's perceptions of bots are relating to something else that we care about,
GOV-BOTs gives us an empirically valid way to assess those internal cognitive states.
Thank you so much for your time and attention today! I hope you enjoy the rest of CSCW. You
can get in touch with me at any of the mechanisms below if you have questions or
interest in collaborating about this work. Bye!