Welcome to 'This is My Architecture'.
My name is Andrea, and I'm here with Saral from Snap.
-Hi, Saral, welcome to the show. -Thanks for having me.
So what do you guys do?
So Snapchat is an app that hundreds of millions of people around the world
use to communicate with their close friends.
Our focus is on close friendships and we enable the fastest way to communicate
by building features like AR lenses, maps, Bitmoji, and spotlight.
Awesome. So today we're going to talk about the platform which resides on AWS.
I'm super excited and let's look at the end-to-end workflow.
Using Snap, sending a snap using Snapchat.
This must be massive scale.
I mean, how many users you have and how many Snaps do you deal with
on a daily basis?
That's a great question.
So we have over 300 million daily active users on the platform.
Okay, that's massive scale.
Let's look at the lifecycle.
You know, I'm a user. I'm using iOS, Android.
What's the flow?
What's the first thing that happens when I send a Snap?
Right.
The first thing which happens is the iOS app uses our gateway,
which is an EKS service, which we have built to talk to the media delivery service
and send the Snap to Cloudfront
and persists it in S3 just so that it's closer to the recipient when they need it.
I see.
Okay, so this runs on Elastic Kubernetes Service.
-How do you orchestrate this? -Right.
So, MCS is the core orchestration service, which we have.
The next thing which the client does is again, the client mix...
Once the client has persisted the media it makes the request to the core
orchestration service, which then checks the Friend Graph.
The Friend Graph ensures that the sender actually has permissions
to send the message to the recipient.
Once the permissions check is done, then it actually persists
all of the conversation metadata into our own database called, SnapDB,
which under the covers, uses DynamoDB.
I see.
So what made you choose to create your own SnapDB as a front-end to DynamoDB?
Yeah.
So we're talking about a lot of scale on DynamoDB.
And we chose to build some higher level features ourselves like transaction, TTL,
as well as how to efficiently kind of deal with ephemeral data in our own layer.
We also deal with incremental state synchronization so that we are not putting
a lot of burden on DynamoDB and it keeps our costs in control.
I see, so, you're storing the metadata in DynamoDB, right, so, level of scale
are we talking about here?
So we are talking about 400 terabytes of data which we store in DynamoDB.
And interestingly, we run nightly scans on DynamoDB,
which run a 2 billion rows per minute,
to kind of do various things on DynamoDB,
including looking for friends suggestions or deleting ephemeral data as well.
So we are very heavy users of DynamoDB.
Okay, so you do nightly scans just to make sure that it maintains
-the relationships and friends and everything. -Right.
Okay, so we talked about the Send Snap.
What about the receiving end?
-What happens? -Right.
So the most interesting part about receiving Snap is, it is very latency-sensitive, right?
Your friend wants to get the message as soon as possible.
So the first thing the messaging service does is, it looks up a connection ID
or server ID from Elasticache to get access to the persistent connection
which a server has to the Android client or to any client in the world
so that we can quickly push a message to that client.
So, we look at the metadata from Elasticache,
we find the server which this client is already connected to
and through that gateway, we basically send the message to the recipient.
I see.
And then you retrieve the data as well, the image or video,
-and chat information through a stream. -Exactly.
So that media ID was persisted all the way through to Elasticache
as well as DynamoDB, so we use that media ID
to kind of get the information from the Cloudfront.
Makes, mostly, sense.
Now this is massive scale, but can you give us
some quantitative details of like what size cluster are you running?
Just give us some numbers.
Yeah, we have over 900 EKS clusters running
and many of those clusters have over a thousand plus instances,
which makes this like a really heavy kind of service graph, essentially.
Makes perfect sense. And I mean, how do you keep this at a low cost, high performance?
I think through, you've mentioned, latency is hugely important on the receiving side but also the cost.
Yeah, for sure.
Like latency when we move to this architecture
we were able to reduce the P50, the median latency
of sending a Snap by 24% which is a huge business impact for us.
Secondly in terms of cost, because we use so much compute footprint,
we are always looking at things like auto-scaling as well as
using the right instant types like Graviton to keep our compute cost look.
That's fantastic.
I mean this is interesting architecture because you're dealing with massive
amount of data
and you're cleverly coming up with innovative ways, for example,
SnapDB, to reduce latency and really make that user experience better
and then use of Graviton and others to keep the cost lower.
So thanks for walking us through this architecture.
Thank you so much.
And thanks for watching 'This is My Architecture'.