Welcome to 'This Is My Architecture'.
I am Hawn and today, I am joined by Daniel from Airbnb. Welcome.
Thanks. It's good to be here, Hawn.
So, before we get into your architecture,
for those who are not familiar with Airbnb, can you tell me a little bit about what you guys do?
Yeah. So, Airbnb is an online marketplace
where guests and hosts can connect with each other
and as a guest, you can, you know, stay in unique accommodations
or have one-of-a-kind experiences with these hosts.
Wow. So, what are some of the business challenges
that you are trying to solve with your architecture?
Yeah. So, at Airbnb, we run our own multi-tenant Kubernetes clusters on EC2.
You know, we want our own clusters because we sometimes need a back port fixes
or add new features into our Kubernetes
and the main challenge that we ran into in the past was,
we have, you know, many different pods that run on a, you know, single host
and we want to give fine-grain access controls or fine-grain IAM roles to these pods.
And so the way we do that now is,
you know, we have our control plane that manages all of our pods in our cluster.
What the control plane does is it injects service account tokens into the pod.
And that control plane is one of the EC2s?
Yes, that's correct.
The control plane runs on a physically separate EC2 host than all of our worker nodes.
Okay.
So, the control plane injects tokens.
These tokens are cryptographically signed, but they contain basically an identity.
They act like an ID card for these pods.
And some things that a token will have are, what namespace the pod belongs to,
and also the name of the pod.
The pod can then take this token and send it directly to STS.
Okay.
You know for verification.
Since, again, since I mentioned before these pods are cryptography signed.
STS can verify that the token is valid and hasn't been tampered with.
If that all checks out, STS can then check the IAM role
and verify that the namespace and the name of the pod
is able to assume the IAM role that it's requesting.
And if all that checks out, you know, STS will get temporary credentials for that IAM role
and send those credentials back to the pods.
So, from a multi-tenant perspective,
how do you secure those pods in a more clustered manner in your Kubernetes cluster?
Yeah, it's a good question.
So, you know, each pod has its own identity
that's, you know, that's provided by our control plane
and you know, in the IAM role definition itself
you know, service owners are able to say like, you know, IAM role Foo, for example,
should only be allowed to be assumed by pods in the Foo namespace.
So you know, given that connection with each other
we can guarantee that other pods and different name spaces
cannot assume you know Foo's IAM role.
Cool.
So in terms of reconciliation and making sure that the pods are assuming the right role.
How do you monitor that and audit that?
Yeah, it's a good question. So you know STS emits events into CloudTrail.
Whenever it creates a new token also whenever our pods, you know, assume an IAM role
those events also get emitted into CloudTrail
and we take these events, and we ingest them into our Elasticsearch cluster
that you know engineers have access to
and sort of the main purpose of this
or the main purposes that we're allowed,
we can, you know, see when a pod is assuming an IAM role.
We can see what IAM roles are being used, we can also see when a failure happens
and when that happens for example, oftentimes, we notice before service owners,
you know, realize that there's a problem, and we can either go, and you know,
check their definition of their IAM role to make sure you know it's correct
or we can let service owners know, you know,
something's wrong and they need to take a look at a problem.
So, you would notify the service owners immediately
for them to correct the roles and configuration.
Yeah and ideally, we have alerting to automatically notify them.
Awesome.
Well, that is so cool you're able to
provide that fine-grain access to the multi-tenancy
and enable it for your Kubernetes cluster, so thank you for that.
Thank you.
And thank you for watching, 'This is My Architecture'.