[MUSIC PLAYING]
CARTER MORGAN: Hi, and welcome back to VM End-to-End,
a show where we have a VM skeptic, myself,
and a VM enthusiast, or VM enthusiasts,
come on to the show to talk about why
VMs are cool, interesting, and useful
for a cloud-native future.
Last time, we talked about reasoning about reliability.
But I feel like I still don't know
enough to actually do anything with it in the real world.
So we've brought Steve and Brian back
on today to help hammer that out.
Hi, Steve.
Hi, Brian.
Thanks for being here.
STEVE MCGHEE: Hey, Carter.
Nice to be here.
CARTER MORGAN: Glad to have you.
Brian, I got to start with you, buddy.
After last week's episode, do you
think I know enough to actually go and like take a small SaaS
offering, and go from three nine survivability to four nines,
to maybe five nines?
BRIAN DORSEY: So I think we've got a lot of the components.
We talked about some of the patterns that are common.
And we can dig into that a little more as well.
But the question is kind of like, when and how?
So what if we got a little more specific here?
Like, what kind of app are you imagining building?
CARTER MORGAN: OK.
Let's go with a payment processing app.
So it's a SaaS application.
And it's going to exist in the cloud.
People do payments.
Right now, that's mostly what it's going to focus on.
I guess it's-- to be honest, I don't even know how to reason
about how reliable that is by itself.
So I guess we should start there.
Steve?
STEVE MCGHEE: Yeah, this is a good place to start.
Like, I think using--
in SRE, we call this a non-abstract systems design
problem, which is the best way to do it.
And I think, if you think about this,
you're building a system on your laptop,
probably, or on your desktop.
You're getting it functionally correct first.
And then you have to think about,
how much do I want to invest in into it in order
to scale it up, to have it do something for someone else?
Because having it just run tests on your laptop
doesn't really make you a lot of money,
doesn't do much good for society.
You want to get it out there, right?
You want to put it into production.
And so what you really have to do is think about, well,
how reliable do I want it to be?
And the answer should always be just reliable enough, right?
That's kind of a jokey way of responding.
But think about who your customers are.
Like, what do they really need out of this?
So you're talking about a payments processor, right?
So like, who are your customers?
Is your customers like your friend
who sells bracelets at the farmer's market,
or is it like a Fortune 500 company who's
going to be like throwing transactions at you all
the time from around the world?
It's going to be very different situations.
You know what I mean?
BRIAN DORSEY: OK.
So I love this specific thing.
So it's always a business kind of tradeoff
or a goals kind of tradeoff, is the takeaway here.
STEVE MCGHEE: Totally.
BRIAN DORSEY: Do you have any like rules of thumb
for how do you know-- because a business thing,
they care about like money and time.
Like, how do you know-- when we're
moving between these thresholds, how much is that?
STEVE MCGHEE: We don't have a lot of good data.
It's more just like a sense.
But the rule of thumb that we use
is that every additional nine you add to a system
is going to cost you 10 times as much as the last one did.
So if you move from 90% available,
just like running on your laptop, and you close the lid,
and it's not available anymore, to moving onto the cloud,
where the VM--
the single VM that it's running on is like 99%,
or actually more like 99.9% available,
it's going to-- it takes more effort.
And the way the reason about it is really like 10x per nine.
So--
CARTER MORGAN: That's huge.
STEVE MCGHEE: Yeah, it's a big deal.
CARTER MORGAN: I like this little simple heuristic for me
to reason about it.
But I guess I'm still a little bit lost in--
like, take the internet.
How reliable is the internet?
Can we ground it, maybe these nines,
in just like a simple like browsing the web kind of thing?
How reliable is that?
STEVE MCGHEE: Yeah, great question.
Have you ever turned off your phone?
CARTER MORGAN: Rarely, but yes.
STEVE MCGHEE: Yeah, like the battery dies sometimes,
or phones kind of lose connectivity.
Like, you ever been in a dead spot?
Like, maybe sometimes, right?
So like, your phone doesn't have 100%
reliability in terms of letting you do stuff on the internet.
So if you think about it that way, like,
if you're building a system just for yourself on the cloud,
you wouldn't want to invest in making it five nines reliable,
because your pipe to that service is over your phone,
and cell towers, and stuff.
And you're going to be driving through a tunnel sometimes.
And so like, why make a five nine
service for a three nines tunnel?
And before you go too far with this analogy,
though, the reason why we start to have higher availability
is because a lot of these services
are actually for server-to-server communication
in a data center, where they have much better connectivity
than you do between your computer
or your phone and the service.
And the other part of it is like, most of the time,
you have more than one customer.
So like, if my connectivity to the service is no good,
most of the other people's service
is actually still good during that minute
or during that second, right?
So you really do want to have, for
like these high-end services, you
want to have them pretty highly available.
But you have to think of the big picture.
Like, who's the customer?
How many of the customers are there?
How often do they expect it to be up or not?
And the last thing you want to consider
is like, your service, or your business,
or whatever, is probably made up of a bunch
of different services.
And so if you have many endpoints
that your service might-- or that your customer may hit,
some of them might be super important.
Some of them might be way less important.
So if I need to sign up for a new account, and it's down,
I can retry, not a big deal.
But if I'm sending you thousands of transactions per second,
and I want you to like give me money back,
that should probably be more up than the signup flow.
You know what I mean?
BRIAN DORSEY: OK, so this makes sense.
So we're building this kind of more reliable system out
of layers of less reliable bits, and because we
have lots of users or business need to do it.
But like, so we talked about a little bit of it.
Like, in the docs, you can find out roughly
how reliable a VM is.
But when you start to build these shapes on top of that,
how do I know even roughly kind of how
reliable a particular pattern that I've built is?
You know, a load balancer in front of a managed instance
group or something.
STEVE MCGHEE: Yeah, great question,
because most interesting businesses,
at least that I know of, are not running on a single VM, right?
They're not just one thing.
And if you look at the cloud offerings,
there's all kinds of stuff.
There's like Pub/Sub, and databases,
and all kinds of stuff.
And your job as an architect, or a designer,
or an engineer at a company who's using the cloud
is to come up with what we call an architecture.
So this is the set of services that all kind of
stick together.
Some of them are like you put code
onto a compute platform of some kind.
Another one is like you put a schema onto a database
or a data store of some kind.
Sometimes it's like a message passing.
So you have to define the message cues.
So this is your architecture.
So the trick with architecture is
you've got a whole bunch of components.
And you can compose them in tons of different ways.
And some of them are complete disasters, right?
Like, some of them do not work well.
So now, the question you ask is like,
how do we know if we're building a good one?
And just like any software development,
the best plan is to use a well-known pattern.
So we have these design patterns in computer science.
We have similar design patterns in cloud architecture as well.
So there's actually a paper that came out sometime
this year, a while ago, and it was
about the-- we called it the deployment archetypes in Cloud.
And it gives you a couple--
like, I think there's six or eight different models.
And it starts off simple, like build one thing in one region.
And it builds up from there.
So some of the more common architectures
are like, if you have a stack of services--
and by services, I mean like your web
server, and your database, and things like that.
You can have one stack in one region, and one stack
in another region, and like a load balancer
in front of those two things.
And you can get really complicated beyond that.
You can have things like failover between those two
be automated.
You can have hot and cold.
You can have hot and warm.
You can have hot and hot.
And you can have more than just two, right?
You could have n different deployments
and have them all be active at any given time.
And you can charge your customers
across them and all that.
So take a look at that paper.
It's not like GCP architecture specific.
It gives you the pattern, though,
that lets you know what you can get out of this archetype.
CARTER MORGAN: I love that.
And I love just learning general principles for how
to architect and design systems.
So I'm definitely going to check that out.
But now this kind of leads me to this question
of calling back something you said earlier,
getting a little less abstract.
If I were to take my payment processor application--
it was just for myself and my friends before.
Now I've got my first big customer.
And they're going to throw thousands
of receipts and invoices at me.
How do I start analyzing going from where
my offering is at to where it needs to be now
to meet these demands?
STEVE MCGHEE: I love pinning it to this real problem.
And the pattern you're talking about,
where you start with a few customers,
and you slowly grow, and you start to make like business
agreements with more customers-- that
is common across everybody.
This is totally how businesses grow.
And it's the right way to do it, because you
don't want to start in your garage
and build like enterprise software
and expect it to be like-- you're
going to spend 15 years building this thing before you have
your first customer.
So growing with your customer demands
is totally the right way to do it.
In terms of actually getting you to be
able to land that contract with, you know, BigCo,
and handle the five nines or whatever
it is that they're putting in your contract,
this takes a lot of what we call resilience engineering
or reliability engineering.
The way to think about it is that you're
going to be building what we call a platform.
And this is going to be a platform of capabilities.
So these capabilities are going to be technical things.
Like, you can deploy code to your services quickly,
and effectively, and safely.
So you've heard of CI/CD, for example.
Like, a CI/CD pipeline is a set of capabilities
in your platform.
Other things, like observability and monitoring,
so being able to determine, what is going on right now?
Why is this one weird thing slow?
Like, being able to look at a graph,
and being able to drill into it, and being
able to check out a trace or a profile
of a live running system, and being able to stand it,
like having your team be able to grok what's really going on
under the covers and use the tools that they
have at their disposal-- this is all part of your platform.
And you're building out capabilities
as a team to be able to run this super complicated, complex
system.
And the more capabilities you have,
the better shot you have at achieving it.
BRIAN DORSEY: So if we want to be
able to handle our requests from kind of BigCo coming in,
everything downstream of that needs
to be highly reliable to the same degree, right?
STEVE MCGHEE: This is a great--
the answer is no.
It's one of these surprising things, where
what really matters to your business
is the front door, right?
So BigCo has a contract with us.
And whenever they send us a thing,
we have to be able to respond within some amount of time.
But does that mean that the database behind our API
needs to be like six nines, and the infrastructure
that database is on needs to be like seven nines?
And then like, does the power supply behind that--
CARTER MORGAN: I've never even heard these nines.
STEVE MCGHEE: --need to be like 19 nines?
Like, what's going on?
This is untenable, right?
And the answer is totally no.
Like, that is not the case.
So this is good news.
So you're able to build things that
are more reliable on top of things that are less reliable.
This is like a general abstraction
that you can achieve.
So if your front door, you want it to be five nines,
you can get away with having like a four nines or even
a three nines back end.
And the way you do that is through some resilience
capabilities.
So like, a really, really simple one
is purely adding retries to your system.
So if your front end is up, and your back end is down briefly,
when your front end makes a request to your back end,
and it goes, woo, error, instead of returning the response
to the customer that you have an agreement with,
like, I don't know what happened.
Like, that's a bummer.
Just wait.
Just wait for 200 milliseconds or some amount of time
and try it again.
If it still doesn't work wait a little longer.
So as long as you're not exceeding
some sort of contractual deadline, you can wait all day.
And you can still give back a correct response.
And the customer will see that as a success.
So this is another form of how we kind of hide errors
behind layers of abstractions.
And it's basically, you're trading off-- in this case,
you're trading off time for reliability.
And if that's OK with your business, do it.
It's just another tool in your toolbox.
CARTER MORGAN: I love this idea of basically planning
for the imperfect.
We talked about this when we talked before.
And I'm sure there's a bunch of different avenues
that we could go and look at this, like geography.
There could be points of failures
there, or like you said, with time, using time as a tradeoff.
The thing I'm really curious about
is, it sounds like there's a lot that goes into this.
You talked about a team of people having to be involved.
And you yourself are an SRE.
So it seems like there's more than just technology.
And I've been very focused on that.
But there's also people involved.
And maybe there's even other elements
involved in this reliability.
Can you talk about that a little bit?
STEVE MCGHEE: Yeah.
When we talk about large systems,
we tend to say people, process, and technology.
So this is a great example of it.
So being able to understand a system, improve the system,
and build the system is-- it's being done by humans, right?
And the model that Google came up with is called SRE.
So a lot of these principles that Google
used to build these systems out and to scale them up and not
burn out all the humans in the meantime
has all been written down.
And the specifics of what Google SREs would
do in the past inside of Google don't apply directly
to what customers would do, because it's
like an entirely different system.
And customers aren't running web search on Borg
or anything like that.
CARTER MORGAN: Right.
STEVE MCGHEE: They're running payments processors
on Kubernetes, on Cloud.
But the principles are actually the same.
And the principles can be applied anywhere.
They're totally consistent.
So all of this bundled together--
not just the tools, but also the process and the culture,
how people work with each other--
we kind of bundle that together as SRE.
And so this is-- there's a bunch of books
out there that you can get that explain all this.
And it's becoming more and more popular.
So--
BRIAN DORSEY: Awesome.
So we'll definitely put some links
to the SRE books and the article you mentioned
earlier in the show notes here.
So we've got-- as far as doing this,
like when you want to actually kind of push your app up
to another threshold of nines, we've got the people, process,
tech part.
And then we've got this reliability patterns.
And some of those patterns, I think,
are built into the system, like we
talked about last time, like are kind of in the VM abstraction.
And some of them are things we have to handle with our app
architecture itself.
Does that sound fair?
STEVE MCGHEE: Totally.
You know, you can-- there's always the idea of
build versus buy.
And a lot of these systems, you're
just kind of getting out of the platform.
So these capabilities that I talked about,
if you're running on-prem on machines
that you bought a couple of years ago,
and you're running your own software on them,
you're not going to get live migration out of it.
You could build it.
It might take a while-- or something like it, right?
But if you, instead, if you kind of take that concern,
and you hand it off to Cloud, you get this capability.
So there's a lot of things out there
that you can get a product, right?
And that gives you that capability for your platform.
And now you can do that thing.
Another similar one is like if you have the a layer 7 load
balancer, a global L7 load balancer,
you're able to siphon traffic from one region
to another, like kind of magically.
It's amazing.
And doing this like on the Wild West internet
is really hard, because it's like you
got to deal with different autonomous systems, and BGP,
and DNS, and all this crazy stuff.
But with Google's L7 load balancer,
it's pretty straightforward.
And you take advantage of this massive set of systems
out there, this front-end technology that Google uses
for its entire set of services.
CARTER MORGAN: I've got to thank you, Steve and Brian.
This is my first time diving into SRE principles
and the thinking behind reliability.
And so I've learned so much, whether it's
about the people, and the process, and the technology,
whether it's about having non-abstract or less abstract
architectures and really thinking through it.
So thank you.
I'm going to definitely check out
that paper about all the different patterns.
And for people listening at home,
if you check that paper out, tell us
in the comments what you liked or didn't like about it.
Thank you so much.
[MUSIC PLAYING]