>> theCUBE presents KubeCon
and CloudNativeCon,
Europe, 2022.
Brought to you by Red Hat,
the cloud-native computing foundation
and its ecosystem partners.
>> Welcome to Valencia, Spain
in KubeCon, CloudNativeCon, Europe, 2022.
I'm Keith Townsend with my cohosts,
Enrico Signoretti, Senior IT Analyst at GigaOm.
>> Exactly.
>> 7,500 people I'm told, Enrico.
What's the flavor of the show so far?
>> It's a fantastic mood, I mean,
I found a lot of people wanting to track,
talk about what they're doing with Kubernetes,
sharing their you know, stories,
some war stories that bit tough.
And you know, this is where you learn actually.
Because we had a lot of Zoom calls, webinar and stuff.
But it is when you talk a video,
"Oh, I did it this way, and it didn't work out very well."
So, and, you start a conversation like this
that is really different from learning from Zoom,
when, you know,
everybody talks about things that work it well,
they did it right.
No, it's here that you learn from other experiences.
>> So we're talking to amazing people the whole week,
talking about those experiences here on theCUBE.
Fresh on the theCUBE for the first time,
Chris Voss, senior software engineer
at Microsoft Xbox.
Chris, welcome to the theCUBE.
>> Thank you so much for having me.
>> So first off, give us a high level picture
of the environment that you're running at Microsoft.
>> Yeah. So, you know, we've got 20
well probably close to 30 clusters
at this point around the globe, you know
700 to 1,000 pods per cluster, roughly.
So about 22,000 pods total.
So yeah, it's pretty,
pretty sizable footprint and yeah.
So we've been running on Kubernetes since 2018
and well actually might be 2017, but anyways,
so yeah, that's kind of our footprint.
Yeah.
>> So all of that,
let's talk about the basics
which is security across multiple
I'm assuming containers, microservices, etcetera.
Why did you and the team settle on Linkerd?
>> Yeah, so previously we had our own kind
of solution for managing TLS certs
and things like that.
And we found it to be pretty painful, pretty quickly.
And so we knew, you know
we wanted something that was a little bit more
abstracted away from the developers
and things like that,
that allowed us to move quickly.
And so we began investigating,
you know, solutions to that.
And a few of our colleagues went to Kubecon
in San Diego in 2019,
Cloudnativecon as well.
And basically they just, you know,
sponged it all up.
And actually funny enough, my old manager
was one of the people who was there
and he went to the Linkerd booth
and they had a thing going that was like,
"Hey, get set up with MTLS in five minutes."
And he was like,
"This is something we want to do, why not check this out?"
And he was able to do it.
And so that put it on our radar.
And so yeah, we investigated several others
and Linkerd just perfectly fit exactly what we needed.
>> So, in general we are talking about,
you know, security at scale.
So how you manage security scale and also flexibility.
Right? So, but you know, what is the...
You told us about the five minutes to start using there
but you know, again,
we are talking about war stories.
We're talking about, you know, all these.
So what kind of challenges you found at the beginning
when you started adopting this technology?
>> So the biggest ones were around getting up
and running with like a new service,
especially in the beginning,
right, we were, you know,
adding a new service almost every day.
It felt like.
And so, you know, basically it took someone going
through a whole bunch of different repos,
getting approvals from everyone
to get the certs minted, all that fun stuff
getting them put into the right environments
and in the right clusters, to make sure that,
you know, everybody is talking appropriately.
And just the amount of work
that that took alone was just a huge headache
and a huge barrier to entry for us to,
quickly move up the number of services we have.
>> So, I'm trying to wrap my head
around the scale of the challenge.
When I think about certification
or certificate management,
I have to do it on a small scale.
And every now and again, when a certificate expires
it is just a troubleshooting pain.
>> Yes.
>> So as I think about that, it costs
it's not just certificates across 22,000 pods,
or it's certificates across 22,000 pods
in multiple applications.
How were you doing that before Linkerd?
Like, what was the...
And what were the pain points?
Like what happens when a certificate either fails?
Or expired up?
Not updated?
>> So, I mean, to be completely honest,
the biggest thing is we're just unable to make the calls,
you know, out or in,
based on yeah, what is failing basically.
But, you know,
we saw essentially an uptick in failures
around a certain service and pretty quickly,
pretty quickly, we got used to the fact that it was like,
oh, it's probably a cert expiration issue.
And so we tried, you know,
a few things in order to make that
a little bit more automated and things like that.
But we never came to a solution
that like didn't require every engineer on the team
to know essentially quite a bit about this,
just to get into it, which was a huge issue.
>> So talk about day two,
after you've deployed Linkerd,
how did this alleviate software engineers?
And what was like the benefits
of now having this automated way of managing certs?
>> So the biggest thing is like,
there is no touch from developers, everyone on our team...
Well, I mean,
there are a lot of people who are familiar
with security and certs and all of that stuff.
But no one has to know it.
Like it's not a requirement.
Like for instance, I knew nothing about it
when I joined the team.
And even when I was setting up our newer clusters,
I knew very little about it.
And I was still able to really quickly
set up Linkerd, which was really nice.
And it's been, you know,
essentially we've been able
to just kind of set it,
and not think about it too much.
Obviously, you know,
their parts of it that you have to think about,
we monitor it and all that fun stuff,
but yeah, it's been pretty painless almost day one.
It took a long time to trust it for developers.
You know, anytime there was a failure,
it's like, "Oh, could this be Linkerd?" you know.
But after a while,
like now we don't have that immediate assumption
because people have built up that trust, but.
>> Also you have this massive infrastructure
I mean, 30 clusters.
So, I guess, that it's quite different to manage
a single cluster in 30.
So what are the, you know,
consideration that you have to do
to install this software on, you know,
30 different cluster, manage different,
you know versions probably,
et cetera, et cetera, et cetera.
>> So, I mean, you know, as far as like...
I guess, just to clarify,
are you asking specifically with Linkerd?
Or are you just asking in more in general?
>> Well, I mean, you can take that
the question in two ways.
>> Okay.
>> Sure, yeah, so Linkerd in particular
but the 30 cluster also quite interesting.
>> Yeah. So, I mean, you know,
more generally, you know
how we manage our clusters and things like that.
We have, you know, a CLI tool that we use
in order to like change context very quickly,
and switch and communicate
with whatever cluster we're trying to connect to
and you know, are we debugging or getting logs, whatever.
And then, you know, with Linkerd
it's nice because again, you know,
we aren't having to worry about like,
oh, how is this cert being inserted in the right node?
Or not the right node, but in the right cluster
or things like that.
Whereas with Linkerd, we don't really
have that concern.
When we spin up our clusters,
essentially we get the route certificate
and everything like that packaged up,
passed along to Linkerd on installation.
And then essentially,
there's not much we have to do after that.
>> So talk to me about your upcoming section
here at Kubecon.
what's the high level talking points?
Like what attendees learn?
>> Yeah. So it's a journey.
Those are the sorts of talks that I find useful.
Having not been, you know,
I'm not a deep Kubernetes expert from, you know
decades or whatever of experience, but--
>> I think nobody is.
>> (indistinct). >> True, yes.
>> That's also true.
>> That's another story
>> That's a job posting decades of requirements for--
>> Of course, yeah.
But so, you know,
it's a journey.
It's really just like,
hey, what made us decide on a service mesh
in the first place?
What made us choose Linkerd?
And then what are the ways in which,
you know, we use Linkerd?
So what are those, you know
we use some of the extra plugins
and things like that.
And then finally, a little bit about more
what we're going to do in the future.
>> Let's talk about not just necessarily the future
as in two or three days from now,
or two or three years from now.
Well, the future after you immediately solve
the low level problems with Linkerd,
what were some of the surprises?
Because Linkerd in service mesh and in general
have side benefits.
Do you experience any of those side benefits as well?
>> Yeah, it's funny, you know,
writing the blog post, you know,
I hadn't really looked at a lot of the data
in years on, you know when we did our investigations
and things like that.
And we had seen that we like had very low latency
and low CPU utilization
and things like that.
And looking at some of that,
I found that we were actually saving time off of requests.
And I couldn't really think of why that was
and I was talking with someone else
and the biggest, unfortunately all that data's gone now,
like the source data.
So I can't go back and verify this
but it makes sense, you know,
there's the availability zone routing
that Linkerd supports.
And so I think that's actually doing it
where, you know essentially,
if a node is closer to another node,
it's essentially, you know,
routing to those ones.
So when one service is talking
to another service
and maybe they're on the same node,
you know, it short circuits that
and allows us to gain some time there.
It's not huge, but it adds up after,
you know, 10, 20 calls down the line.
>> Right.
In general, so you are saying that
it's smooth operations at this very,
you know, simplifying your life.
>> And again, we didn't have to really do anything for that.
It handled that for us.
>> It was there?
>> Yep. Yeah, exactly.
>> So we know one thing when I do it on my laptop
it works fine.
When I do it with across 22,000 pods,
that's a different experience.
What were some of the lessons learned
coming out of Kubecon 2018 in San Diego?
I was there.
I wish I would've ran into the Microsoft folks,
but what were some of the hard lessons learned
scaling Linkerd across the 22,000 nodes?
>> So, you know, the first one
and this seems pretty obvious,
but was just not something I knew about
was the high availability mode of Linkerd.
So obviously makes sense.
You would want that in, you know
a large scale environment.
So like, that's one of the big lessons
that like, we didn't ride away.
No.
Like one of the mistakes we made
in one of our pre-production clusters
was not turning that on.
And we were kind of surprised.
We were like, whoa, like all of these pods are spinning up
but they're having issues,
like actually getting injected and things like that.
And we found, oh, okay.
Yeah, you need to actually give it some more resources.
But it's still very lightweight considering,
you know, they have high availability mode
but it's just a few instances still.
>> So from, even from, you know,
binary perspective and running Linkerd
how much overhead is it?
>> That is a great question.
So I don't remember off the top of my head, the numbers
but it's very lightweight.
We evaluated a few different service missions
and it was the lightest weight that we encountered
at that point.
>> And then from a resource perspective,
is it a team of Linkerd people?
Is it a couple of people?
Like how?
>> To be completely honest for a long time,
it was one person Abraham,
who actually is the person who proposed this talk.
He couldn't make it to Valencia,
but he essentially did probably 95% of the work
to get into production.
And then this was before, we even had a team dedicated
to our infrastructure.
And so we have, now we have a team dedicated,
we're all kind of Linkerd folks,
if not Linkerd experts, we at least
can troubleshoot basically.
And things like that.
So it's, I think a group of six people on our team
and then, you know
various people who've had experience with it
on other teams. >> But others,
dedicated just to that.
>> No one is dedicated just to it.
No, it's pretty like pretty light touch
once it's up and running.
It took a very long time for us to really understand it
and to, you know, get
like not getting started,
but like getting to where we really felt comfortable
letting it go in production.
But once it was there,
like, it is very, very light touch.
>> Well, I really appreciate you stopping by Chris.
It's been an amazing conversation to hear
how Microsoft is using a open source project.
>> Exactly.
>> At scale, it's just a few years ago
when you would've heard the concept of Microsoft
and open source together
and like OS, just, you know--
>> They have changed a lot in the last few years.
Now, there are huge contributors.
And, you know, if you go to Azure,
it's full of open source stuff, everywhere so.
>> Yeah.
>> Wow. The Kubecon 2022,
how the world has changed in so many ways.
From Valencia Spain, I'm Keith Townsend,
along with Enrico Signoretti.
You're watching theCUBE,
the leader in high tech coverage.
(upbeat music)