[MUSIC PLAYING]
BRUNO AZIZA: Welcome to another episode of "Data Journeys."
This is where we come to learn from data leaders.
And today, I'm really excited to talk to Jack, Jack from Reddit.
Thanks for taking the time to talk to us today.
JACK HANLON: My pleasure.
Really looking forward to it, Bruno.
BRUNO AZIZA: So, Jack, you are the VP of Data at Reddit.
You are a very experienced data analytics leader.
You were the VP of analytics at Walmart.
And in 2009, you even were a data analyst at Google.
So you've done a lot with your career.
Let's start with the basic what you're doing today.
What does Reddit do?
JACK HANLON: Reddit's a community of communities.
It's a place where individuals find experiences
built around their interests, their hobbies, their passions.
And what we're all about is about,
how do we bring community and belonging to the world?
And that's always been a mission I've
been incredibly excited by.
But when you look around today, that
feels like a particularly important thing for us to do.
BRUNO AZIZA: That's excellent.
So 100,000-plus communities, 52 million daily active users.
So that's going to lead you to a lot of data.
So let's talk a little bit about these use cases.
What are you trying to do with data?
How are you innovating with data today inside Reddit?
JACK HANLON: Yeah.
Well, with that kind of traffic, you
can imagine the volume is wild.
And so we're ingesting north of 55,
probably closer to 60 billion events a day.
And from a technical standpoint, how do we make sense of it?
From a business standpoint, though, which
is where the magic is made here, as I'm sure you know well--
from a business standpoint, the question
is, how do we make decisions better, faster?
So that's internally in the company.
How do we make decisions better, faster
about the things we need to do in terms of strategy,
the choices we need to make?
And then on behalf of users, when
we think about things like recommendations
and data-powered products, how do we
make decisions better, faster to help them find communities
more effectively and faster?
BRUNO AZIZA: And so there are a few things here.
One, you have a lot of data.
You've also grown tremendously, 2x growth.
You're also preparing for 10x growth.
So you know you've got even more data, and it's kind
of compounding on itself.
And you're also launching 1,000-plus experiments per
year.
So how do you manage that complexity?
And what are the specific use cases
across anomaly detection or user data
that you're driving to make sure that this data is
serving the organization to drive bottom-line growth?
JACK HANLON: Yeah.
You mentioned some great examples
in terms of experimentation.
Obviously, for a product like ours,
when we're hitting so many millions of people,
we need to understand the impact of everything that we do.
We have a huge responsibility as a social network
to think about, what is our impact on the world.
And experimentation and a really close sense
of, for all that volume coming in,
what data is meaningful, what data is true,
and how we interact with that and how
we react to that is-- it's unbelievably important
to be able to serve that key role.
In terms of anomaly detection, I think
you're flirting with a data quality conversation
there, which, as you might imagine,
takes up a good part of our headspace at any given time.
Because we want to say, how do we
make decisions better and faster,
but how can we possibly do that unless we
know that the data is legitimate and that it's completely there?
BRUNO AZIZA: Can we talk a little bit
about how you're using data to serve
particular communities inside Reddit, like your employees
specifically?
If you think about sales, marketing, finance
and how they're taking advantage to the such massive data
to make better decisions, what are some of the top use cases
there?
JACK HANLON: Of course.
So we mentioned experimentation, which
is just a subset of the overall class of things we might
need to use for product data.
So every ads team, every product team, they're
thinking really extensively about how we use
data to inform our strategies.
And I want to be clear that I'm saying "data-informed."
I think "data-driven" is a great buzzword.
But from a vocab perspective, we want
to use data to inform our decisions
and think about our overall strategic backdrop.
There are sometimes decisions we want to make
that when we have the data, we're saying, look,
I know the data says this.
But we know this is the way that we want to take the company.
That's OK as long as we're looking at it
and making that conscious choice.
BRUNO AZIZA: That's a great best practice,
just thinking about the difference
of informed versus driven.
Let's talk about maybe an experiment where you're
particularly proud of the experiment, how
it was rolled out, and the department that used the data.
What kind of decisions did they end up making?
JACK HANLON: We're running so many at any given time.
Hard to dial in on a specific one.
But I think recently, we are rolling out
a new class of algorithms and personalization
for our home feed.
And this is a huge deal for Reddit.
Historically, the home feed has just
been an aggregation of your subscriptions.
But what we're doing here is two things.
One, we're starting to surface more of your subscriptions,
the communities you're subscribed to.
We're starting to surface more content
for more different communities.
And we're also starting to insert recommended content
that we think will be interesting to you, which,
obviously, isn't a novel idea on the web more broadly
but in the Reddit context is quite different.
And so you might imagine our initial concern
might be, how will some of our older users respond.
We've had some users who've been with us for 15 years.
How will they respond to a core change like this?
But guessing and worrying about that
isn't great to run a business.
And so, instead, when we're running these tests,
we actually were able to see both from quantitative and
qualitative feedback that our older users are loving it.
The engagement is higher because people are talking about
they've-- they're seeing subscriptions.
If they're subscribed to many communities,
they're seeing some content that they haven't seen in years.
And on the recommendations front,
they didn't think there was more content for them to discover.
They thought they had their curated list,
and yet here they're finding new things to engage with.
And it's keeping the site fresh for them.
It's keeping the experience fresh.
And so that's incredibly exciting for us
and is a great example of where you
can choose to worry about risk, or you
can choose to really lean into opportunity thoughtfully
and have the information at your disposal to make a good choice.
And we are incredibly excited about the gains
we're seeing in terms of people subscribing to new communities,
engaging more with the app, engaging more
with different content, posting more content.
It's all been very positive.
BRUNO AZIZA: So everything really
is directed towards making sure that you're
engaging the community, providing them
with fresh, relevant content, maybe
helping them discover things that-- communities they should
join, users they should engage with that they might not
have thought about before.
JACK HANLON: Absolutely.
And by the way, doing so, though,
with an eye also on safety metrics.
I think it's very easy to sort of hack your way
into engagement growth.
But I think we all have seen where that
can become really dangerous.
And so the other thing I think we're really
thoughtful about and very proud of is, in our data,
we're making very clear any potential negative impacts
to communities and to people.
And those things absolutely stop us
from rolling out any of these changes.
But we've seen great results there as well.
These are safe, positive mechanisms
to increase engagement.
BRUNO AZIZA: So now that we've talked about engaging, what
about internal communities?
I'm thinking more about your internal teams
and how they leverage data to make better decisions.
What are some of the use cases that you're
dealing with here where you feel like you've innovated?
JACK HANLON: Well, when we're thinking
about helping people make decisions,
we have a lot of different places where that occurs.
Obviously, there's, internally, the strategic level.
How does the organization make great decisions
about what we're doing?
There's also decisions as they relate to the consumer, so
things about how we generate engagement
or how we create more community or how
we get people to create more communities
and create a space where they feel comfortable doing so.
And then, obviously, we have advertisers.
And so when we think about advertisers, how do we actually
run experiments and help advertisers understand
the impact of running on our platform
and see the value of being able to reach our many communities?
All of these areas, we need systems
that are able to talk in a consistent way
to different audiences to help them make
different types of decisions but do so
in a way that shares a common sense of truth.
And at the end of the day, if we can solve for that,
then we've done a great job.
But obviously, as you know, that makes it
sound a lot easier than it is.
BRUNO AZIZA: Let's talk a little bit about your best practices
here because we've talked about the use cases.
Now we want to drill into what you've
learned, positive and negatively, throughout
your experience, not just at Reddit, but at Walmart
and other companies before.
So maybe let's start with the positive.
What's the piece of advice you want
to give everyone that you must absolutely start with or never
ignore?
JACK HANLON: One thing that's been consistent from Reddit
and back into Walmart and other places
is privacy and security have to be a top-level concern.
It is much easier to focus on how
a system works in terms of its speed, its responsiveness,
its uptime.
And you can put out great stuff technologically there.
But these aren't neutral systems.
Once you're carrying private information,
once users trust you to offer private information,
either the things they talk about,
the communities they participate in, or in previous lives
the things that they purchased, that is precious information.
And taking that seriously and having
that be a top-level concern, it's unbelievably critical.
And so for us, the way that that is actually manifested
is we've built a specific team, which is the user data team.
And that team is made up of data scientists,
data engineers, other back-end software engineers.
So we've made it so it can be a contained team that
can get things done themselves so they can move with speed,
and they can move with clarity.
And they're creating the access for all user
data for our other teams, for our advertising targeting,
for our consumer ML teams, for any team that
needs to access user data, consistent ways of accessing
it, safe ways of accessing it that
only offer what that system needs and nothing extra.
That investment has been incredibly crucial for us.
And it allows us to operate, actually,
with more speed because we don't need to guess.
We don't need to onboard other stuff.
We have consistent ways where if I'm
building new systems somewhere in the company,
I know, OK, I just talk to user data.
I get the data I need, and my service will work.
BRUNO AZIZA: That's an amazing best practice.
People often talk about privacy and security.
But it's difficult to think about, OK,
how do I operationalize this.
And in your case, a central user team
with data scientists, with analysts
that are enabling the rest of the company
sounds like something that's really working well for you.
What about data quality?
Often, we talk about data quality.
I'm sure you're dealing with so much data,
you've got to go through that yourself.
How did you tackle that problem with such a large volume
of information?
JACK HANLON: We could spend all our time just
talking about data quality.
It's such a huge topic because, ultimately, all of the things
I talked about before, none of them
work unless data quality is working.
And some earlier tech companies feel
like they don't have to deal with this as
much because services are new.
Reddit, at 15, we have some legacy systems,
and we have new systems.
So this is a real concern.
But honestly, anyone with big volume
has to be thinking about this a lot.
And for us, I think it was a combo of ideas.
There's both, every team needs to take on the onus of this,
to some extent, and say, I'm responsible for the quality
of the data that my system outputs.
Because data is so valuable, we want
to treat how the user experience or the advertiser experience
or things like that works as being of equal importance to,
what is the data that comes out of it?
Is it high quality?
Is it the data that we need?
Because ultimately, they're equally important to us.
So we can say that, and that's fine.
But we need to create tools, then,
that make it easy for those teams, tools and processes that
make it easy for those teams to actually operationalize that.
So sand testing tools, a lot of mechanisms
for people to test new features before we're pushing things out
to the apps, those are critical features there.
But then also, look, everybody's going to make mistakes.
So we need to have great anomaly detection.
We need to have great rules and systems
to be able to interpret.
Are we seeing the right things come through?
And then, even after all of that, that's great.
OK, now we've convinced people they need to do it.
We've given them the tools to be able to do it.
We're catching the stuff that's falling through the cracks.
But look, for companies that have been around for longer,
there are surfaces that are entirely unowned.
So what happens when a surface like that
starts having a data quality issue?
Or what happens when a very core top metric that
is above what a lot of other teams are working on
needs adjustments?
So for us, that would be something like screen views
or time on site.
In that case, we also have staffed
a team who's focused on being a SWAT team of sorts
on jumping into these situations where it needs direct support
and we need to make sure it's done right,
because we can't afford to get into a two-month-long
discussion with six teams to figure out
who's going to fix this thing.
It's just got to be right.
We have to be able to have this information
correct, accurate, fast to be able to make decisions.
It's a competitive space.
And it's a space where we want to provide
great stuff to our users.
BRUNO AZIZA: So this is a great example of both best practices.
You have both the technology implementation,
but you also have the team to enable
you to actually operationalize the changes that are required.
What about the opposite?
What are some of the mistakes that you've
witnessed throughout your career that you really
want to warn people and say, just avoid
these things because they'll get you in trouble?
JACK HANLON: I think that some people think
of data as being adjacent to software engineering,
in some ways.
And that's, in part, because there are
some specialties in each area.
But in reality, the things that I think
are most problematic in most data teams are
those where they're ignoring some great software
engineering best practices.
So one of those is trying to make
one service do too many things.
I think there's a huge temptation in scope creep
to have--
you have some huge service that's
doing a ton of ingestion and cleaning.
And, oh, there's anomaly detection.
Oh, and you can-- it throws off nice reporting
about what's going on.
And there's different access for different types of users,
and there's governance.
And at a certain point, you're going
to release this thing in four years,
and all the technology's moved on.
And you needed all these things earlier.
So splitting this up into individual services
and having the ability to iterate through each
of these things, having the ability
to really understand what each smaller service is outputting
and inputting to be able to see when you have issues,
how you fix those things--
maybe not the most novel advice in the world in the software
engineering context.
But I still see data teams make this mistake so frequently
of wanting to have one ring to rule them all.
And that's a dangerous game.
BRUNO AZIZA: I think it's very useful because I think,
as technology evolves, I think it's very tempting to bring
in all these services and building
this gigantic system that ultimately becomes
unmanageable and probably is just
not effective at doing what you want
it to do in the first place.
I think it's actually really, really useful advice.
What about other advice that you might
have that you would think you see people making mistakes
left and right on this?
JACK HANLON: So the other one is a little bit
about relationships.
And I think a lot of data teams take
the role of just inheriting the requirements
from other teams of saying--
another team says, I want to start this service
to do thing x.
Here's the level of access I need, the latency, whatever.
And a lot of I see a lot of data teams saying, OK, great.
Let's do this.
But in reality, sometimes those upstream teams
are not actually well-positioned to make some of those choices
and can add a tremendous amount of complexity
or a tremendous amount of development time
to services that probably aren't helping the business overall.
And so I'd say a really critical idea here
is knowing when to step in and more heavily influence
other teams' decisions when it comes to data.
And so sometimes the right answer is, you know what?
You need streaming, and it needs to be this amount of data.
Sure.
We're going to find ways to make that work.
And other times, it's going to be,
hey, you want us to be able to do this,
but it's going to take two months of extra work
to ingest that last 5% of events.
And that's crazy.
That's crazy.
We all have such important work to do.
You actually need to change your system to throw off
only these new types of events.
Or you need to change the system to adhere
in these various ways.
I think that conversation--
I don't hear it as frequently as I
would hope between teams about, what are the ways where we need
to adapt, versus, what are the ways where
we need to set the standard.
And having a really healthy balance of those things,
it creates a much faster overall engineering organization.
BRUNO AZIZA: Interesting best practice.
In the world where we have more and more technology,
what I'm getting from you is people
make decisions, collaboration matters,
discipline around engineering.
And so I think the foundational pillars are still
very relevant.
Jack, I feel like I've learned a ton from you.
Use your teams, your SWAT teams around data quality,
how to not forget the basics of engineering--
those are great lessons for our audience to learn from you.
Thank you again so much for taking the time
to talk to us today.
If you want to find out more for stories just like this,
click on the link down below.
Until next time, I'm Bruno Aziza.
[MUSIC PLAYING]