Hi there, I’m Dr. Michael Ann DeVito, I’m an HCI researcher, a transgender woman,
and a content creator on TikTok I’m here to tell you about how transfeminine
people like me get attacked when we try to help our communities
How algorithmic systems make it much, much worse
And how we can help fix this issue through design
But first, I want to talk about one of the creators in my study
Participant 3 is a 27 year old transgender woman with no specific media or advocacy training
Who decided to get on TikTok, throw on some cat ears
and start providing hope, health information,
and a living example of how trans people can be happy, and successful, and goofy, and loved
She puts a lot of time and effort into this, and when I asked her why, she said something
I found to be heartbreakingly familiar: I just was so miserable living as a boy, that
I was like, “fuck it, I'll take all of the negative to not have to do this anymore,”
and the fact that there was positives kind of blew me away.
I wanted other people to be able to see that.
That’s why Participant 3 puts the time in – to help people.
And it worked She grew a huge community
She became a role model She literally became the reason some other
folks felt safe transitioning And then came the slurs
The attacks The coordinated reporting campaigns
The death threats The direct harassment of her followers
Participant 3 chose to serve her community by becoming publicly visible as a transgender
woman And she used TikTok to do it
her reward?
Trapped in a morass of harassment, abuse, and hate speech
I call this dynamic “The Algorithmic Trap of Visibility”
The algorithmic trap of visibility holds that trans people are offered algorithmic doors
to visibility, made up of affordances and features which
can help boost one’s visibility Those algorithmic doors exist in a context
where online visibility in particular is more important than ever for trans people.
online spaces and communities are how we connect with each other,
advocate for ourselves and provide material aid for those in our
community who need it The benefits of those spaces directly depend
on folks who have opted to pursue the kind of visibility
we’re talking about To take on the risks of becoming visible to
educate, to welcome,
to protect, and to help
However, in the algorithmic trap of visibility, every algorithmic door to visibility is also
potentially a trap The traps are the design and policy decisions
that determine what’s ”acceptable,” From the overall fact that you have given
up ultimate control of who you are visible to
to a system that does not share your values, your sense of scale,
or your particular safety concerns as a transfeminine person,
To content moderation policies that don’t count transphobia and transmisogyny
as hate speech and leave transfeminine creators without a
useful way to report harassment and abuse And yet – we keep posting, because we know
those benefits of visibility are crucial!
To keep achieving our goals in this kind of environment,
we find algorithmic trapdoors to evade these now-scaled,
far more dangerous algorithmic traps We come up with strategies and tactics
Which we can use to avoid, modify,
or counter the unwanted outcomes of allowing our content
and identities to be mediated by an algorithmic system
And that strategizing is powered by Folk theorization A folk theory
is an intuitive, informal theory that an individual develops
to explain the outcomes, effects, or consequences of some aspect of the world
Here, folk theories are how we figure out how to navigate algorithmic environments
Even though platforms refuse to tell us how they actually work
They’re how we construct useful guesses And ways to think about what might be happening
to our content and why And how we start to think about how we might
affect those outcomes ourselves – What algorithmic trapdoors we might slip through
if abuse and harassment start coming our way To figure out which algorithmic trapdoors
we can try and escape through, Transfem creators start by building an overall
folk theory of how TikTok forms audiences and distributes content
And here that mostly takes the form of the “expanding stages” folk theory.
This is participant 4’s version of the expanding stages theory, which I asked her to draw out
for me: First, the post goes out to your followers
If the video is a dud with that first group of viewers,
it pretty much only gets distributed on the following page -
its time on the for you page is done If the video does well,
It winds up on the for you page of folks who TikTok somehow thinks are likely to follow
you if shown the right content Then, if you do well there,
You go to the general for you page - you’re at the beginning of some level of
virality Importantly, this version of the theory also
asserts that TikTok frequently makes mistakes during
this process When tiktok gets it right,
It sends you to an audience that is genuinely receptive to your content,
That wants to watch, enjoy, engage, and learn from it
In a positive, constructive way It sends you to what most creators I know
call “your side” of TikTok Where folks like you
And content like yours Are welcome
But when TikTok gets it wrong It sends you to the “wrong side” of tiktok
Where there’s plenty of engagement to be had
But it is not constructive engagement it’s weaponized engagement:
It’s harassment It’s hate speech
It’s threats It’s coordinated campaigns of abuse
And false reporting That can destroy useful, important communities
This is how the trap of algorithmic visibility works on tiktok,
In all versions of the expanding stages folk theory
Transfeminine creators are offered a door to visibility:
Make content that folks engage with And engage with those folks in turn
And we’ll make you visible to wider and wider audiences
Allowing you to do more and more good But there’s a trap:
It just keeps happening If things go well for you,
the system keeps distributing your video to more and more people like the people that
are engaging and that’s great
But if things turn around on you and suddenly most of the engagement is harmful and abusive
the system keeps distributing your video to more and more people like the people that
are engaging and now that’s horrifying
and traumatizing But transfem creators persist and engage in
folk theorization along the lines I’ve described in my past work
Forming folk theories like the expanding stages theory
Which can act as frameworks for coming up with novel, strategic algorithmic trapdoors
Let’s take a look at how that plays out using comments as an example
Comments on TikTok can get very nasty, very fast
As participant 11, a trans woman who makes content about being transfeminine and jewish,
illustrates: “It definitely started with a few comments
from more conservative people that got responses from my more left-leaning audience,
and then it really blew up over the course of two days.
And then there was a 24 hour period in which it was it was being shown to neo Nazis…
fights started happening in the comments, people would say some shit…
That kind of engagement really seemed to spur the algorithm on.”
Working within an expanding stages folk theory, Participant 11 thought through the situation,
Deciding that conversations not only drove visibility
But specific kinds of conversations with specific kinds of people
drove visibility with those specific groups That theorization then informed new versions
of her strategies for moderating and responding to comments
For both herself and moderators on her lives No more engaging with the trolls for Participant
11 or her followers, because correcting antisemetic, transphobic
hate looks the same as agreeing with it to the platform
Participant 11's folk theory of TikTok allowed her to figure out what to do to protect herself
Without sacrificing her goals She didn’t have to do something like turn
off comments, Which would cut off the very discussion and
community that she’s trying to create in the first
place So, what can we do to help transfem creators,
and other marginalized creators navigate and overcome the algorithmic trap
of visibility?
First, many of the problems creators talked about
Get much worse when distribution gets big and spirals far out of the creator’s control
or expectations We need a way for creators to slam the brakes
on content distribution when they see things taking a turn for the
worse without doing something like making a video
private or deleting it entirely which destroys existing visibility and often
precludes future positive visibility I’m proposing that platforms like TikTok
implement an algorithmic emergency brake that would allow a user to remove a piece
of content from algorithmic distribution overall or, better yet, based on set thresholds, such
as rate of distribution to non-followers Second, most of the other problems creators
talked about came down to decontextualization:
Decontextualized negative interactions driving distribution to hateful audiences
Decontextualized language triggering moderation systems to
mute and block the victims of attacks instead of those who were clearly the attackers
We need better mechanisms for user-side contextual input to algorithmic systems,
We can consider interventions as simple as extended input fields in the reporting process
so moderators can get more information on why content might be something like hate
speech in that specific context As well as more complex solutions such as
a system of creator-side flags which can be used when interacting
that would let a creator literally tell the system
that their comment was just rebuking transmisogyny or racism
and not meant to be read as a cue to bring in a new audience of transmisogynist racists
There’s a lot more in the paper, including Algorithmic community, identity flattening,
and cultural context folk theories a discussion of algorithmic paternalism,
and an extension to my folk theorization framework Introducing actionable and demotivational
folk theories