Hi, I'm Matt from AWS.
And on today's episode of 'This is My Architecture',
I'm joined by Keith from Pinterest.
Thanks for joining. Thanks for having me.
So tell us about Pinterest.
Pinterest's mission is to bring everyone the inspiration to create the life they love.
And that requires processing a lot of data.
The data engineering team at Pinterest hosts a variety of data engineering services
like Hadoop and Spark and Presto, all on their EC2 instances.
That's great.
I mean, earlier in my AWS career, I did a ton of big data analytics work.
So I'm really excited about this talk today.
So I know that we were talking before the shoot
and there's sort of a before and after story here
in terms of how you give your users internally access
and services, access to data and associated clusters
to sort of process and analyze that data, is that right?
Yeah, yeah.
So we have over time, teams at Pinterest have come to us and said,
Hey, I've got this data set,
and I want to restrict access to this data set to make sure that I know how it's being used.
But I also still want to have access all of the services like Hadoop and Flink,
and I need to be able to build and train models off of this stuff.
So how can you go and give me access to that?
Okay, so what did you do sort of before?
So before what we typically do is like each of these teams
for your sake of illustration, we'll just say that each of these teams
has like a single AWS bucket, a S3 bucket here.
So we'll have S3 bucket. That's S3 bucket.
Yeah, bucket one, and we'll create bucket two.
Okay.
And what we would normally do is create an IAM role
for each of these.
That grants access to the data in that bucket. Exactly.
So each of these IAM roles pairs with a bucket in this particular example.
So if we go with this traditional model,
what we end up doing is including this IAM role into a variety of different services.
So we build out a Hadoop service, a Presto service,
Airflow, all of the different systems that we count on inside of Pinterest.
And so what you can see is in this model, I've basically built out two different copies of Hadoop.
I have to configure them, I have to maintain them.
The scale there is kind of difficult.
So that's because the instances in the Hadoop cluster, for example,
I have to assume the role Exactly.
and then to get access to the data.
So if you need to have a different role, you need a different cluster. Exactly.
We've used sort of the traditional model of assigning the EC2 instances to the IAM role.
Yeah. And as a result, they're paired.
And every time I create a new IAM role in this model, I end up having to create a new set of services,
which also means that if, for example, some customer comes along tomorrow
and needs to pull data from both bucket one and bucket two
I actually now need to create a third IAM role.
And a third set of my service is just to maintain that.
So here I've only got two different groups of data,
but I've already got three different sets of services just to maintain access to them.
Yeah. I can see how that could get untenable in a dynamic company like Pinterest,
as needs change over time, that's going to mean more clusters, more cost, right,
and really more roles you're going to have to maintain.
Yeah, so we were really looking for a solution that we could scale
and start adding more and more of these data access groups.
So every team could come up with whatever they wanted to do to maintain their own data.
Okay, so we sat down with the Pinterest security team
and with AWS folks, and we started talking about the Assume role API call.
We were already using the assume role API call with a specific IAM role,
and then would give us back an STS token,
which we could then use for a variety of different services,
that could then communicate directly with, for example, S3 buckets.
But what we didn't really know about, and then we started learning about
was that you could actually pass in a list of managed policies
that would become session policies for that STS token
and those managed policies get ended or intersected with the base IAM role.
Okay, so fast-forward to today, I guess what's over here,
you have STS, and you're calling assume role, right?
Yeah, we're calling assume role.
And so we're passing in this base IAM role. Yeah.
And right now this base IAM role we're giving permission to access both of these two buckets.
Okay. And then we're going to create two other managed policies.
We'll just call them managed policy one,
and we'll have managed policy two, - yeah.
- down here.
So let me call assume role if we were to pass in the base IAM role
and just managed policy one, even though the base IAM role
has access to both of these buckets,
the actual STS token produced we'll only have access in this case to manage policy one
and thus the first bucket.
But if I pass in both of these managed policies
that resulting STS token will have access to both of those buckets.
So I can build out all the permutations with just two managed policies here.
Okay. Well then how does it know what managed policies?
How does it connect to internally what the user should have access to?
Yeah, so that's where we built a token vending service on the Pinterest side.
And the token vending services job to sort of bridge our own internal concept
of users and authentication with these managed policies.
So the token vending service is going to get a call in from customers
or from EC2 instances
that are running services down here and make a call-up to our LDAP service.
And it gets back from that LDAP service, a list of LDAP groups that that user is in.
So at Pinterest you use LDAP internally for all of our users - Exactly.
- groups that they're in.
Okay, got it. - Yeah, so each user might have one or more LDAP groups
that are related to this.
So we can map those LDAP groups
each individually to sets of managed policies.
In this particular example, if a user makes a call into the vending service
and is in LDAP group associated with managed policy one,
they get a token that only contains managed policy one,
they only have access to the first bucket.
If they make a call in with both LDAP groups, they get both managed policies,
they get both buckets, or they could get just bucket two.
And then if the group they're in changes, that change immediately takes effect.
Yeah, so one of the cool things for us is that all of our tools
that we use to internally administrate LDAP groups all apply here.
So I can actually make the team that owns each of these S3 buckets, own their LDAP group.
And they get to choose who has access to their data and not.
That's great, because it's pushing ownership down to the sort of the service team owners really.
Right, it makes it really easy to maintain.
And so the way this works from here is the token vending service,
and if we consider, for example, Hadoop here that we're hosting on our EC2 instances,
Hadoop has a credentials' provider that we've plugged in there.
The credentials' provider is going to take a kerberized Hadoop call
So the user authenticates gets a...
Yeah, so the user authenticates provides a strong authentication token
through Kerberos that authentication token gets sent to the token vending service.
The token vending service takes the identity out of that.
After authenticating it passes it up through LDAP to get back to managed policy
group mappings and then produces an STS token.
And from there the EC2 instances now have a token that they can use
to directly read from S3.
Now is this only for humans or can use it with like with software with other services as well?
So while we do use this for user accounts, we often use this for workflows and services that are offline.
Those services run as LDAP service accounts.
It still goes through all the same systems.
They're still using the exact same managed policies and LDAP groups.
So they're actually getting the same provision configuration mechanisms.
That also means that when I'm granting access to a service account or say a given workflow,
it's now explicitly getting access to the data in the LDAP groups that it's being paired with.
I love it.
So you've really created a general purpose token vending service that can be used by humans
for Hadoop clusters or for any number of use cases, really.
Yeah, yeah, it actually accepts in a bunch of different authentication types.
So we've plugged in OAuth and other things.
So it plugged into a variety of different services.
It's about a dozen different services Pinterest has plugged it into so far.
So ultimately this is a simple, scalable solution
that will save you some money because fewer clusters to run like in the old model.
There are fewer IAM roles to manage.
It's really a great architecture.
Thanks for sharing it with us.
Thank you.
And thanks for watching 'This is My Architecture'.