Welcome to "This is My Architecture."
I'm Matt from AWS.
We're in Las Vegas on the third day of AWS re:Invent.
And I'm here with Paul from Skyscanner. Thanks for joining.
Hi there.
So tell us about Skyscanner?
All right, yeah. We're a travel search company.
And we're based in the UK.
And we have about 80 million unique users on.
Wow, 80 million.
So you need a lot of infrastructure to support it.
We do.
We have a lot of infrastructure to support that.
It makes your job fun.
- All in AWS. - Awesome. Thank you.
So I see you have a Kubernetes cluster up here.
I was thinking about this video series.
And two years ago, roughly when we started,
a couple of customers were talking about Kubernetes and containers,
but it was still kind of getting started for a lot of them.
Fast forward to this video shoot
and seems like every second video we're doing is about containers
and about Kubernetes, and EKS, ECS.
So I guess it's good to see, and I think a lot of our viewers
are going to appreciate the tips you give them today.
Yeah, absolutely.
Yeah, there's definitely a lot of buzz about containers at this re:Invent.
So the first thing I noticed is, you know, this is Kubernetes cluster,
and we were talking before
and you were saying you're running this on EC2.
So we have EKS,
and, you know, there's other ways you can run Kubernetes,
a lot of ways you run Kubernetes in the cloud.
Why did you do it this way?
Yeah, we did this way because we were maybe six to seven months before...
- Ahead of the curve. - EKS was even announced.
So I think that we made a decision to move to Kubernetes
just before EKS was public knowledge.
- Okay.
So we decided to deploy ahead ourselves.
Okay. And I see here you have sort of...
These representing Availability Zones or...
Yeah, so what this represents is the underlying infrastructure
for a typical Kubernetes cluster at Skyscanner.
And what makes our infrastructure slightly different
is the fact that we're running on Spot.
- Spot Instances. Okay.
- 100% on Spot. - 100% Spot? Okay.
Yeah, that's going to be interesting for a lot of people.
Yeah.
I see ASG, so those are Auto Scaling Groups
spanning multiple Availability Zones, isn't it?
Yeah, so I mean, so it's an individual ASG,
being deployed across multiple AZs
as a fairly normal setup for Kubernetes cluster.
However, what we do is we have multiple ASG.
So we have two here, but typically we have five ASGs.
And what that gives us is,
it gives us diversification in the Spot fleet.
So diversification of instance type?
Yes, of instance type.
So we're on c3s, c4s, c5s,
m4s, r3s in our clusters.
Great.
So I guess with diversification, with Spot so that way
if there's not enough available capacity for one, you can use another one
Yeah, well with Spot, you've got pools of capacity pair per instance type, per EC.
Yeah.
The more unique pools of capacity you pull from,
the less likely that you'll lose all of your capacity in your cluster
at any one time.
And what kind of scale are we talking? Are these really big instance types?
You said 80 million users, how does this translate?
- Yeah.
So we run multiple clusters across multiple regions.
- Okay.
Within an individual cluster, we are potentially
sort of 120, 130 nodes at peak.
Typically, sort of see 8x, 9x cluster instance types.
Okay, so c5, c4s, and m5s - the big ones.
So you're talking maybe 3,500-4,000 cores at peak.
Yeah. Okay. Wow. Okay.
And how many sort of queries you're getting?
What kind of volume is this supporting?
Yeah.
So we typically at peak on a single cluster in one of our busier regions,
it will be between 60,000 to 70,000 queries per second.
Wow.
Okay, so 60,000 to 70,000 queries per second, 80 million users,
that's a lot of volume.
I imagine handling that is not for the faint of heart.
So how are you sort of handling scaling with Spot,
100% Spot in your Kubernetes cluster?
Yeah.
So I think it really goes down to diversification.
And because we have multiple AZs
with multiple and different instance types
across multiple clusters and multiple regions,
we're essentially broadening out that pool of capacity.
And because we're using AWS infrastructure
in the actual managing of the instances
is really handled by the ASGs.
And the cluster autoscaler is the piece
that's manipulating the desired count on multiple ASGs.
And the cluster autoscaler, it refers to an ASG as a node group.
That's how it collects instances of the same type together
and then presents that back
to the rest of the Kubernetes infrastructure
to meet it's...
- So when you said desired count, you mean desired count of nodes of EC2 instances?
Yes, yeah. Within an individual ASG.
Okay, great.
So is there anything that you think viewers would find interesting
or might learn from your experience
using 100% Spot in this case for Kubernetes cluster?
Yeah.
So I think the first thing you have to deal with
is the spot termination.
- Okay. - So that's something you can't avoid.
So we have a shorthand script
which will cordon the node
so it prevents the Kubernetes scheduler
from actually pushing more pods...
- So when you get that two-minute termination notification,
how does it actually know that the node is going to be terminated?
Yeah, so every five seconds there's an R script
which runs and checks the metadata.
The instance metadata? Okay.
Instance metadata, it sees the termination notice,
it then kicks off the script to cordon the nodes
and then it deregisters that from the ELB
and then it drains the node completely.
Okay.
And how does it actually prevent new traffic
from going to the node?
So our setup is slightly different in that we run our ASG proxy,
ingress controllers across all the nodes in the cluster
that gives us quite a high degree of resilience
in terms of inbound traffic to the clusters.
- Okay.
So we simply unregister the instance from the ELB
that stops traffic coming in from the outside.
Okay, great.
So you handle both like the unregistrations from the ELB,
but also like Kubernetes is aware that the notice...
- Yeah, we have to tell the ELB to stop sending traffic.
But we also have to tell Kubernetes to treat this node
as being unavailable and to stop scheduling
work until we can
gracefully drain the services that are still running on it.
Are there any gotchas with this approach with taking nodes out,
that you've learned along the way?
And not specifically.
I mean, I think the different pieces, the fact that we have ASG proxy instances
running on all of our nodes,
we have to have this extra step to drain the ELB.
And but apart from that
its just standard Kubernetes commands that were running.
Okay.
And did anything, as you were building this,
have not work as expected, or did you have to...
Yes.
We encountered a feature with the cluster autoscaler,
which prevented it from autoscaling
when a single ASG reached zero.
- Okay.
The cluster autoscaler relies on at least one node,
being the autoscaling pool, to describe it, to actually,
figure out what type of resources it has available.
And it's centered on edge case where it can handle that,
and we got into a situation
where we had to restart the cluster autoscaler every so often.
- Okay, how did you fix it?
And we applied a patch, which is,
which was great.
- Okay. - So that solved that problem.
Well, okay, that's a great tip for viewers watching out there
who might be restarting their clusters.
- Yes. Yeah.
And so I guess one more question.
We were talking about your use of Spot,
which I think is great in autoscaling groups.
Some user, some viewers watching this might be thinking,
why aren't they using reserved instances instead?
Why the Spot approach?
Well, we felt because of the diversification factor,
the multiple clusters, the multiple regions,
we could take advantage of the full saving
that we could get from Spot.
- Because they're so much cheaper? - Yeah.
They're considerably cheaper,
and they're considerably cheaper than reserved instances as well.
Well, so you're actually able to get it, like cheaper than reserved instances.
- Okay. - So it was cheaper. Yeah.
Overall that is what has been cheaper.
And it also gives us the flexibility,
you know, to use different instance types.
And this is, we're relatively new to this level of scale in Kubernetes.
So we still didn't feel,
we, I guess, had seen all the different types of workloads
that we have in these Kubernetes cluster.
So using reserved instances,
we might have picked it on instance type,
and that would just have...
It would have been harder for us
to deal with different workloads as we move forward.
Okay. Well, that's great.
I think a lot of viewers watching this it's going to resonate with them,
the fact that you're running at such scale across multiple regions,
you know, not just multiple AZs, multiple regions,
and that you're really using 100% Spot.
I think it'll help people overcome that initial barrier sometimes
when they think about...
- Yeah, I mean, I think that's the thing with Spot
is the actual technical solution wasn't that complicated to roll out,
especially with support from the AWS SAs.
I think it's more about understanding the risk
and approaching it from that sort of perspective
and making sure that you're comfortable with the risk
and you have ways to mitigate it.
I mean, we have run books to switch back to on demand,
if that becomes a problem.
Okay. It's always good to have a backup plan.
Yes, it is.
Well, great. I really like this architecture.
I think a lot of people will appreciate it.
- So thanks for sharing with us. - Welcome.
And thanks for watching "This is My Architecture."