- Let's test you on policy a little bit.
- Okay. We'll see how little I know about policy.
- Do you think that if a video has somebody
using a slur against a protected group of people,
marginalized group, that you,
we should take the video down?
- Yes, absolutely.
- Okay. Well what if the slur is being used
as part of a comedy routine?
- I guess I would have to see if it was mean spirited
or if it was more humor based.
- What if the slur is being used
by somebody who's actually a member
of the group that the slurs about, that be okay?
- Yeah, I think absolutely.
- Okay. What if you're a reviewer on a different continent
and the slur is about someone's religious faith?
How do you know if that person is, that's using the slur
is a member of that group?
- Yeah, no, you wouldn't.
I guess you'd have to.
Yeah, that's a really good question, I don't know.
- Yeah, and I mean like what if
they may not know someone's faith or orientation?
It's a lot harder than people think it is actually
to draw some of these policy lines
and then be able to enforce 'em consistently at scale.
- Hello Insiders.
I'm Rene Ritchie, your creator liaison
and I am sitting here today with Matt Halprin,
Global Head of Trust & Safety at YouTube.
How are you Matt?
- I'm good. Nice to be with you, Rene.
- So you did a blog post recently
where you went into how policies are made
and enforced on YouTube.
One of the first questions we get,
I think it's really foundational
is why does YouTube have to have policies at all?
Why can't it just be if the content is illegal,
we take it down and that's basically it.
- Yeah, yeah. We do get that question a lot.
Since the earliest days of YouTube,
we have had community guidelines and we want to,
you know, balance who YouTube is,
which is really to be an open
and a place for freedom of expression.
But balance that against harm to users,
and to the ecosystem of creators,
and just being responsible in general.
It might be one example of that is adult content.
If we had no policy against adult content,
it would probably overtake the site.
And that's not what, you know,
everybody wants YouTube to be.
And so we have a policy
against certain kinds of adult content
and then you start extending that to other places.
Like, what about child safety?
You know, should it be okay to, you know
to do something that actually harms children?
You know, we don't want that.
And then what about things like hate or harassment?
Do we, you know, creators actually don't want
to be harassed, it turns out,
nor do other people want to be harassed.
And so while we have a pretty high bar
for what meets our you know, bar for removal for harassment
or for hate or something, we do think it's important
to have policies that cover these areas to,
you know get that balance
between freedom of expression and user safety, right?
- One of the things that we hear quite often
is that the policies are sometimes
really hard to understand.
- So I don't actually think each individual policy
is necessarily that hard to understand
but like all of them together, that's a lot.
I mean it just, it is, it's a lot for everybody.
We have tried to work on improving our help centers for that
although generally people read the help center
only when they violated their policy.
And that's normal, I think.
We listen very carefully to this feedback
and we have been doing some work in the background
with our product team, you know,
considering are there some adjustments we should be making
in how our strike system works, for example,
and you know, no promises on exactly how or when
or what or whatever like that,
but that's because we're listening to this feedback
on things that we might be able to do that, you know,
could help people understand and, you know,
make it the purpose of the,
like the strike system is to be more about education,
and so are there any things we can do
to improve the policy experience
and help people be able to understand our policies
before it impacts their account?
- I think sometimes also for creators there is,
for example, trust and safety, like Community Guidelines,
but there's also Ad-Friendly Guidelines.
And then there are some things like age-gating,
which can be applied across several different things.
- So, you know, look, our platform is designed
for, you know, age 13 and up.
We talk to child safety experts
who point out the 13 year old mind
is different than an 18 year old mind,
is different than a 50 year old mind,
and different kids are different.
We have done a number of things,
but one of which is we do have age-gating,
where we say it's okay for this content
to be on the platform, but it may not be appropriate
for somebody under 18 to see.
That's not really any different than what happens
in, you know, the TV, movies, video games, things like that.
I mean, that's, we're just kind of
lining up with them there.
Of course, the issue is when something gets age-gated,
it also gets demonetized.
When a video gets age-gated, it gets demonetized,
and we understand that that can have
a significant impact on a creator.
We appreciate that that's frustrating and we would like,
if someone wants to avoid that,
then they need to kind of, you know,
keep their their content a little bit different
so it doesn't get age-gated,
but we think it's important to be a responsible platform
to make sure that we're not showing something
to a 13 year old that maybe they wouldn't see, you know,
at a movie or in a TV, on TV or, you know, in a video game.
- Then how do you balance creators
who might want as few restrictions as possible
with other groups who might want more restrictions
or with different countries that have
either tighter or looser cultural norms?
- Boy, that second part is really tricky.
Cultural norms around the world
are not exactly the US' norms,
which lean very strongly towards freedom of expression,
and so, you know, when I travel to my teams in Asia
or Europe, it's common to get questions about, you know,
"Don't you allow too much on the platform?
Aren't you too permissive?"
But we do have a strong freedom of expression ethos,
and that is something that we take around the world.
We don't have different standards, which makes it hard,
it makes it hard for people in those markets,
for creators, but that's, you know,
it's not really feasible to have a different set of policies
for, you know every single country.
With respect to creators,
some creators wanting more or less, you know,
truthfully if oh, you know,
half the people think we're too restrictive
and half the people think we're too permissive,
we might have that policy in about the right place.
- Something else creators
worry about a lot, especially creators
from historically marginalized communities,
is that flagging by itself, especially mass flagging,
could take down their video, their channel.
How does that work?
- You know, it doesn't matter whether there's one flag,
you know, 1,000 flags, 10,000 flags,
it literally makes no difference
into how the evaluation works
or whether content should stay up or stay down.
The first thing to say is people can look
at our community guidelines enforcement report
and they'll see that over 90% of the videos we remove
are actually caught by our machines first,
and then they'll be sent to a human to take a look at
and validate it or not.
So that much, more and more technology is the one
that's catching these more than user flags,
and again, there's nothing relevant
about whether it's 1 flag or 10,000 flags.
- That raises an actually really interesting question there
because I think sometimes creators worry about bias
or subjectivity or just personal opinion
rather than an official policy when they're being reviewed.
- The process is something, is basically
a content moderator gets served up a video,
and they're evaluating it against a policy,
and the enforcement guidelines behind that policy,
and they're just comparing one against the other
and saying "Does this cross the line or not?"
And then we evaluate those raters frequently,
within a biweekly basis
to see how their quality performance is,
how their accuracy is,
are they implementing our enforcement guidelines accurately
our policies accurately?
They compare the content they take down against those
and see whether or not it has tripped the line or not.
If they take something down they shouldn't,
then that's a ding against their score,
and the opposite, too.
If they leave something up that they should have removed,
it's the same thing
and they're just getting measured against that.
So that helps keep any potential bias
out of the process.
- Sometimes creators feel it's like a speeding ticket,
you know, they get pulled over
while they watch a bunch of other cars just race by,
so their video comes down,
they see another video that they feel is very similar
that's still up.
- There's a few reasons why that may happen.
One is, is simply just time.
You know, we have 500 hours a minute of video being uploaded
and so we may get to one video first,
and another one may show up the next day
or something like that,
and so there's time period in between.
Second is what seems like the same video or the same clip,
the videos may actually get evaluated differently.
So let's say that, you know,
one creator makes a violative comment
and or there's a clip shown of somebody speaking
and they just show it as a kind of a raw thing
for people to look at,
and it comes down 'cause it's violative.
The second video has the same clip,
but in that video the creator actually says, "Oh
and this isn't true."
Or, you know, "Of course we don't believe in something
like that", and that's what what might qualify
for an ETSA exception, an educational, documentary,
scientific, or artistic exception,
where, because they have refuted the violative claim
or condemned the violative claim,
they're just kind of exposing it for that purpose.
That video may stay up
even though the same violative clip gets in there,
one exception is there, one's not,
this happens oftentimes in news coverage,
where there'll be analysis about it
and explaining why it needs to be refuted.
The third is sometimes we just miss it.
I mean like sometimes we just miss something
and that we hope some people will flag it to us
so that we can actually get more consistent
and, you know, take the right action.
- One thing we've seen regularly is policy updates.
Why do we update policies?
- The world happens on YouTube,
and the world keeps changing,
and norms keep changing.
We see new types of content,
and we take a fresh look at something,
and we realize, you know, when we wrote
that policy three years ago, we didn't really contemplate
this kind of new content that's coming up
and I think we need to make an update.
Industry standards changed.
You know, what, because norms change
and so what used to be, you know,
a Teen is now a Mature or used to be Mature is now a Teen.
Same thing on, you know, whether it's TV or video game,
and so we try to align with industry standards often
and so we've made updates recently on both those areas,
especially in the adult content area.
- That does create a little bit of anxiety
in the creator community, though,
because we never know if old videos,
like, we can't count on old videos still being viable
in a new world and sometimes it's hard
to understand why new policy should apply to older videos.
- This is a tough one and I hear you for what you said
and how frustrating that would be to have a video
that you upload at one point and was compliant,
and all of a sudden is violative,
and so it ends up getting taken down.
Almost every time we change a content policy,
if we end up having to take down old content,
we don't give you any kind of account penalty.
So we'll remove the content,
but there'll be no strike issue to the account
because we don't want to adversely impact a creator
when we move the goalpost,
as opposed to the creator didn't know our policy,
or didn't abide by the policy when they uploaded it.
We often give a grace period,
it might be 30 days or something like that
for people to adjust to the new policy
before we'll start issuing strikes.
We may again remove,
but before we would issue any kind of account penalty,
you know, why do we take it down?
If norms have changed
and we thought it was important enough to update a policy,
then we actually think that content shouldn't be on YouTube
independent of the date that it was uploaded.
We just try to mitigate it
and how it impacts the creator's account.
- Yeah, I guess because a new viewer will come on
and still see that content as if it's fresh and new to them.
- Exactly.
- Sometimes it's also a feeling in the community
that when policies change
or when we decide to address something
we've heard from creators,
it takes a really long time for that to roll out.
Can you give us an idea of why that is?
- Once we do all the research
and we go through our internal processes,
and we agree on a new policy line,
it goes through a number of steps
before we put it live.
And that's because we want to be consistent
when we get it out there.
So the first thing we do is the people directly on my team,
we'll give them 25 videos and say "You guys
go apply the new policy to this."
And maybe the first time they do that,
they only match 70% of the time,
and then we give 'em 25 other videos and they get,
and eventually they get to like 95% or higher,
or a 100% match rate.
That's great.
Then we take that and we give it to a bunch
of our content moderators, but not all,
just like a small group and we say, you know,
we give 'em the training materials
and say "How do you do?"
And they may start out being 70% right,
and eventually through multiple cycles
of updating our training materials,
maybe even modifying some of our enforcement guidelines
that we thought were clear but aren't actually clear
then we kind of eventually they get
to like a 95% consistency rate.
At that point, and only at that point we will go out
and train thousands and thousands of people
because we want it to be consistent
when it gets out live to everybody
and that process can take weeks and even months.
- That brings us to timestamps,
which is something that we hear a lot from creators,
that it's really helpful when they can tell
exactly what part of their video
was being actioned by policy.
I know we're making more and more
of those available to creators,
but do you have any sense of where we are now with that?
- Yeah, well the first thing I would say
is you're preaching to the choir
because part of my job is I'm just a content moderator.
Like the high profile decisions we make
often end up at my desk before we actually, you know,
render a final verdict.
And if someone on my team
sends me a video without a timestamp, I'm not happy.
And so I can totally get it.
You know, creator should get timestamps too,
if we're gonna take a video down.
And so we have been working on that for a couple of years.
We've rolled it out to a number
of different policy verticals
and our goal is to get it to to all policy verticals
as soon as possible.
It's just, it's a matter of time,
but I would hope that by, you know,
maybe the end of '23 or early in '24,
we'll have most of those covered.
- Matt, thank you so much for spending this time with us.
I know like policy, trust and safety,
all of that is deeply concerning to creators,
and we really appreciate the transparency
- Anytime. We appreciate everything creators do.
- And thank you all so much for watching.
If you have any questions, comments,
just please leave them below.