Welcome to another episode of 'This is My Architecture'.
Today I have here with me Thom from T-Mobile.
-Hi Thom. -Hi.
So Thom, we all know T-Mobile.
So tell me a little bit of what your team does.
I work in a team that does central planning for cloud.
We have hundreds of teams, thousands of developers with T-Mobile
and many of those teams that use the public cloud, use AWS.
So what we do is we build platforms, security tools, and other services
to enable those teams to build our customer-facing features on AWS.
I see. And those tools that you build for them,
do you have a standardized way of deploying them?
Yeah, so we will build, for example our Kubernetes infrastructure.
We have a standard way of building Kubernetes infrastructure
where we will use our automation tools like Teraform...
and Ansible.
And basically what we're building when we do that
is we're building Kubernetes clusters
with making sure we have our identity management baked into
the clusters we're building.
And the 20 to 30 different components that go into building a Kubernetes cluster.
And what this does-- so this is the automation stack that we use.
What this does is it allows us to have a standard way
to build Kubernetes clusters
across the entire organization.
So anyone who is consuming Kubernetes is doing the same thing.
We don't have drift in terms of
the number of different components people are using,
the versions, or the state, that sort of thing.
Okay, so basically they can just take what you build
and deploy as many times as they want
depending on what applications they are developing.
Correct.
We have several applications, for example, our e-commerce application,
our retail applications, our internet service provider applications
that use this system with hundreds of applications that use this system
and they're all deploying to a standard infrastructure
across all the applications.
Okay. Let's dive a little bit deeper
into the architecture of this standard infrastructure.
You mentioned Kubernetes.
What else are this team using?
So, the ability to build these common clusters
allows our team to focus on the infrastructure
and allows the application teams to
be able to consume things like RDS for example.
So teams don't have to spend the time
on thinking about what infrastructure they're building.
We're using a GitOps model for example.
And so any change to the state of any cluster anywhere,
when there's 40 clusters now running 30,000 containers
across hundreds of applications,
any change to any of those is done through a GitOps process
which is just a merge request for example.
And so that has a review process and it goes out to production or non-prod.
Okay, so the infrastructure deployment is
completely automated through Github and Teraform.
And the infrastructure itself,
do they have the same components?
Each team uses the same components in the infrastructure
or do you have different components depending on the needs of the teams?
So from a component point of view,
what we're trying to do is we're trying to build,
effectively we're trying to get to the point where we have
immutable infrastructure.
And that's really the goal.
So we've been at this a long time.
We've been doing this for almost five years now
where we've had large scale container based environments.
And we've evolved to the point now where we want to basically just say
that a cluster is not something that is living.
That you deploy to it when there's configuration change needed.
You remove it and redeploy through this GitOps process
to deliver immutable infrastructure.
So this is kind of the end state of where we want to be.
The enables us, for example, to take all these applications we have.
Like for example, each of these applications lives in a VPC.
This VPC sill span east and west regions.
So we get HA out of this.
So we really want to make sure everything is consistent across these.
We'll also have prod versions of these and non-prod versions.
Dev versions of these as well
so that we can have consistency in every singe environment that someone uses.
We'll also make sure that we can have a single way to build clusters
even though we have different needs for different clusters.
We have dedicated clusters for teams.
We have share where many teams are in one cluster.
And so we have clusters that span accounts.
Or, clusters that span VPCs.
And all these things need to be baked into
how we deliver our code for our infrastructure.
And all of these, let's say dedicated share accounts,
do they share information at any time?
Are they connected to the VPCs at any time?
Yeah, everything is done through Transit Gateway here.
We have a model where we have about 100 accounts or so,
which is a fairly small number for a company our size
and the scale we're at.
But we're able to do that because of tools like Transit Gateway.
Transit Gateway allows us to connect into our data centers.
So we have data centers in Texas and Washington state.
And so that allows all these applications here
to connect back to any information they need
from the data centers.
And so we can replicate this process across many different accounts,
many different use cases.
So you're deploying this across your data center, regions.
Are you deploying also closer to your users?
One example, one good benefit of having a model like this
is you can take something new, like a local zone for example,
and we can use exactly the same process to deliver out to now the LA local zone.
So in LA you have a new local zone that we can deliver to
and that has to AZs.
And all these clusters we built all span three AZs.
And so we want to be able to do the same thing here
where you have an A and a B on LAX
and deliver this.
We were able to do this--
Now, local zone is different than a region and so you have to adjust for that.
But we were able to make those adjustments and deliver
a local zone within a week.
So it was pretty straightforward for us to be able to look at that.
So what was the reason to actually see the deployment on a local zone?
Do you have any applications that need single cycle latency?
Yeah, so as a telecom, we really understand the need for low latency
and to build applications where customers can access things quicker.
But on Amazon today, we use west and east.
So we need more options. We need to be able to
service our customers in LA, Florida, and they're not exactly close to a region.
So one of the ways we can do that is through local zones.
So if you can think of for example, a store locator application
where every time somebody is looking for a store,
they're looking for something near to them.
So that will help us make that a quicker process.
So, you already deployed on local zones?
Was there any difference or any improvement
on the latency when you did that?
Yeah, it was significant.
We'd been measuring the difference between local zones.
So a user in LA going to a local zone will experience pretty low latency
in the area of five milliseconds is what we've tested.
But going back to the AZ will be something like 50 milliseconds.
So it's an order or magnitude faster.
I see. Just the last question I have for you
is all of this infrastructure,
I'm assuming that you are, besides the automation,
also constantly monitoring it.
Yeah. So obviously we hook into CloudWatch.
We pull our CloudWatch into this infrastructure as well through Prometheus.
And we have...
Observability is one of the key points to being able to do this at all,
never mind doing it at scale.
And so this is something we really leverage
and make sure we have all the alerts firing
and we know the state of our applications, or infrastructure across AWS
and across all of our clusters.
This is very interesting. Thank you so much sharing this with us.
Sure. Thank you.
And thank you for watching another episode of 'This is My Architecture'.