DALE MARKOWITZ: Hi, I'm Dale.
This is Making With ML.
And today, I want to talk about something
you've probably experienced if you've ever
been on the internet.
I'm talking about user-generated content that violates
a platform's terms of service.
[MUSIC PLAYING]
Almost all content platforms, whether they're
YouTube, or Twitter, or OKCupid, or "The New York Times",
have some strategy for dealing with bad user input.
And this is called their moderation strategy.
Now, you can imagine what's OK to post on Reddit
or a dating app might not fly on "The New York Times",
so these platforms have to consider what is allowed
under their terms of service.
These platform maintainers also have
to consider the implementation of their moderation strategy.
Let me give you some options.
Option number one-- you could hire a team of human moderators
to go through every piece of user-submitted content
before it's posted to OK it.
This is called pre-moderation.
And as you can imagine, it's really good at making sure
that nothing bad slips through the cracks,
because human beings are actually
looking at what's being posted.
But it also doesn't scale very well,
and it takes a large human moderation
team that scales with the size of your user input.
So you'd only use it if it's extremely, extremely important
that no bad bits get through the cracks.
Option number two-- you could ask your users
to flag bad content for you and then send this flagged content
to a human moderation team for later review,
maybe in a sort of priority queue
where they figure out what's important to review first.
This is called post-moderation, because the content
is moderated after it's already been posted.
For example, when you're served a YouTube video or a Google ad
and you have the option to report that content,
you're seeing a post-moderation strategy.
The advantage to doing this is that it
allows your human moderation team to scale.
But the disadvantage, of course, is
that then your users have to actually see this potentially
offensive content.
And in some instances, this would be unacceptable.
Both of these approaches have major drawbacks.
You're either having your human moderation team go through
every single piece of content before it's posted,
or you're potentially subjecting your users to content that
should have been flagged but that wasn't.
And neither of these approaches work for real-time moderation,
like for example, if we're doing a video stream or chat
messages, which brings us to option number three--
AI-powered moderation, which is what we're building today.
The idea is to use machine learning
to analyze content like, text, speech, images, video, and flag
the content the instant that it's created.
You could then send that content to a team of human moderators
so that your users never have to see it,
but your moderation team can scale better.
Now you might wonder, why not use
AI to just moderate your entire moderation solution?
So let me add a disclaimer here.
Machine learning models are always probabilistic,
which means that they invariably make mistakes.
And sometimes they're biased, meaning
that they make mistakes in particularly bad, consistent,
and potentially embarrassing ways.
So I always recommend using AI in conjunction
with human beings, having this human in the loop.
For example, you might have your algorithms put comments
into a moderation queue for your moderators
to review, or allow users to appeal decisions
that were made by the AI.
So knowing that, today I'll show you
how to make a simple AI-powered moderation bot for the chat
platform, Discord.
Let's get started.
Today, we're going to build a bot that
sits in a Discord server and analyzes user messages.
When the bot detects a message is toxic,
it will flag it with a dynamite emoji.
It'll also keep track of insulting,
flirtatious, spammy, and nonsensical messages and flag
them with an appropriate emoji.
It'll keep a scoreboard of emojis or karma
for each user in the chat room.
And if a user types too many toxic messages,
they'll get kicked from the room.
Now, don't worry if you've never done any machine learning
before because, for this project,
we're going to use the Perspective
API to do the complicated machine learning bits for us.
It takes in strings of text and produces scores
for different attributes, like toxicity, insultingness,
explicitness, and even just incoherence.
To create these models, the Perspective team
use millions of comments from data sources,
like the New York Times, Wikipedia,
and many other open data sets.
If you want to see how they built
the model and the architecture behind it
and even learn how to train your own,
check out the Perspective API website.
To use Perspective, you need to fill out
a quick Google form explaining how you plan to use the API.
Once your project is approved, you
should be able to use it in your app.
Next, go ahead and download the Making with ML GitHub repo
and navigate to the Discord moderator folder.
Here's all the code you'll need to run your bot.
The file, Perspective.js shows you
how to use the Perspective API in JavaScript.
At the top, you can see all the different types
of attributes the tool recognizes,
like insult, toxicity, spam, incoherence,
flirtation, and more.
Now, let's connect the Perspective API
to a Discord bot.
To do that, you'll need to sign up for a Discord developer
account.
Once you have an account, create a new application.
Then create a new bot by clicking
on Bot in the left-hand panel.
Give your bot an avatar and a cute name, and you're set.
The brain of our Discord bot lives entirely
in the file Discord.js.
Here at the top, you can see our emoji map,
which tells our bot how to react to different user comments.
Flirtations get a kissy face, and spam gets a fish.
Down in the Evaluate Message function,
we call the Perspective analyzeText
function to detect attributes in the message.
If an attribute is found, we react with an emoji.
And if the user has sent too many toxic messages,
we temporarily kick them from the channel.
The next step is to run your Discord bot from the command
line.
Awesome.
Now, it's ready to be added to your server.
In the Discord developer panel under OAuth,
select the permission your bot needs--
sending messages, adding reactions, and kicking users.
Copy the URL above into your browser.
And from here, you'll be able to add your bot to a server.
Now if we go to our Discord server, boom.
The moderator bot is in the channel,
keeping track of everything I type and reacting with emojis.
If you want to build this Discord moderator by yourself,
the code is in GitHub.
And you can follow along in my very detailed blog
post in the description.
And there you go.
Now you have your very own AI-powered chat bot
moderator for Discord.
Now, the moderation strategy that our bot used
is very simple.
It just kicks users from the channel
when they say too many things the AI thinks is toxic.
But like we discussed earlier, that's
actually not a very good strategy
for a real in-production moderation,
because the model can make mistakes.
In fact, the Perspective website specifically says, this API
is not designed for fully automated moderation solutions.
And they even list a bunch of known biases
that the model has.
Just think of it this way.
AI is just one piece of the puzzle,
and it works better when human partners are in the loop.
That's all for now.
If you want to learn more about moderation and AI,
make sure you check out the blog below.
And let me know what problems you want to see
solved with AI in the comments below.
But remember, those comments are moderated, too.
That's all for now.
See you next time.