How can one service take down so much?
Hi, everyone. Leo Notenboom
here for Askleo.com.
This video is based on an article that was originally written a couple of years ago
in the wake of a large cloud service provider's downtime.
And the side effect was that a bunch of other services stopped working.
They weren't explicitly linked to the cloud service,
but it was clear that because of the downtime they were based on or they
were using that cloud services technologies.
And since I'm now using that technology myself, I wanted to explain exactly
what's happening in the cloud and why so much of it is happening.
The service in question is Amazon Web Services.
When they built their bookstore so many
years ago, they realized along the way that they were building some pretty
impressive and pretty unique computing infrastructure.
They decided that in addition to selling
books, they would also rent time on their equipment.
That grew and grew to the point that I
think AWS probably makes more money for Amazon than selling stuff does.
I could be wrong.
It's a big part of their business, though,
and as a result, what that means is that a lot of places
that you think are isolated, there are services that are just doing
their thing on their own computers that happen to be connected to the Internet.
They're doing their own thing on Amazon's computers, on AWS equipment.
Yeah, Ask Leo! Is one of them.
AskLeo.com is hosted on an Amazon Web Services server.
If Amazon Web Services go down, I go down.
But there are pros and cons to doing that, things that a lot of people have thought
through and evaluated, especially in the light of the occasional
hiccup from Amazon or from any of the other large cloud providers.
There are a lot of benefits to the approach.
The one for me that I just really
appreciate is that it's incredibly flexible.
If today I suddenly realize that my server is too small or too slow, I can,
with a setting change and a reboot, have a bigger, faster server.
In fact, I've done it.
Ask Leo today
is running on a slightly different configuration than it was, say,
a couple of months ago, specifically because I wanted it to be faster.
It's generally cheaper.
The pricing is really interesting at AWS.
They don't charge you per month, they charge you per minute.
So if I fire up a server and run
an experiment that lasts me maybe 30 minutes, I pay for 30 minutes.
In other words, they pay less than a buck or something like that.
It depends on the configuration.
The bottom line is that makes it very lucrative to host there,
because if you make a change, it's only for the duration of the needed change.
When I resized my server, for example, I did it in the middle of the month.
The first half of the month was the old price.
The second half of the month was the new,
more expensive price for a slightly more powerful server.
A lot of companies take advantage of this
because they can do that much faster than I just described.
For example, in my case,
if I decide Asclio needs to be on a bigger server, I need to go to the interface.
I need to shut down Asclio, make the changes and bring it back up.
It's down for a few minutes with a properly written application,
an application that was written
specifically for Amazon Web Services that could all be transparent.
In other words, if I chose to do so and it's no small task, it will get me wrong.
But if I chose to do so,
instead of taking Ask Leo down to make a change to the one server,
I could instead say, oh, traffic's getting too high.
Let's bring up another server.
I could have a whole raft of servers all servicingaskleo.com. Now,
as you might imagine, that's complicated because of the way
that data would have to get spread across the multiple servers.
But a lot of the large online services that you're already using do exactly that.
What seems like a single machine, a single computer that you're interacting
with, is potentially a bunch of these different virtual servers that are being
managed physically in the Amazon Data Center and managed
logically by people at the companies you're using.
Now, of course, there are risks, right?
The risk we saw from the downtime is
that you've put all of your service on somebody else's computer.
All of your fortunes are now tied to their fortunes.
So, yes, like I said, the risk I run, the very real risk I run is that if
Amazon Web Services goes down, Ask Leo goes down.
If I lose access to my Amazon account,
for whatever reason, I no longer have access to my servers and services.
Essentially, they go down.
So the bottom line there, though, is that that needs to be considered.
There are two things, especially for a small guy like me.
There are two things that come into play.
One is that I'm small potatoes, right.
If Amazon Web Services has a problem,
there are some really big sites that are going to be impacted.
And you know that AWS is going to be moving and scrambling as quickly as they
can to resolve whatever it is that's causing the problem.
When your service going down makes national news headlines or sometimes
global news headlines, you can imagine there's a lot of attention
being paid as to how quickly you can get that repaired.
So I'm happy to ride that coattail.
They wouldn't move that fast for just me,
but they would absolutely move everything they possibly could because they are such
a large organization, and an impact like that impacts so many people.
On the small side, though,
sometimes the failures aren't system wise, sometimes they're not global news.
Sometimes it really is just about me
this is why I have backups from the day I started to ask Leo.
To be honest, from the day it started
on its own web server, one of my goals was always to be able
to reconstruct a recent version of the website
on a completely different host without needing anything from the original host.
So any backups that might be on AWS, any backups, any data that might be on AWS.
If that were gone, I would still be able to find a new host, bring up the website,
and get close to what it was before it disappeared.
That's a lot of work, but it's the second level of safety net, right?
If Amazon has problems, I'm going to rely on them to fix it as
much as I possibly can, because they're going to do it much faster.
On the other hand, if I have a problem and it somehow
involves leaving Amazon, then I periodically try to make sure that,
yeah, I've got all the pieces somewhere else so that I can do exactly that.
Now there are alternatives to Amazon Web services.
Google has a huge cloud service.
Microsoft has Azure.
They're big cloud services.
There are others.
Cloud computing is ubiquitous, and because of the scale,
because of the size of these endeavors, they're typically much more cost effective
than certainly having your own server that you have to manage,
or even having a virtual server in a smaller data center by some other
third party that you may still or still not have to manage.
But it all really does come down to risk management.
All of these services, all of them including the one you might
run yourself if you've got your own server in a closet, there's risks.
There's always going to be risk.
That means that the right decision is not avoid the cloud.
You can't.
If you've got a server on the Internet, you're in the cloud.
What you want to do is make sure that you've got the safety net in place.
That allows you to recover from the kinds of things that can go wrong.
As is one of the overarching themes of Ask Leo.
That all begins with a backup.
Now, the one little
update to the original story of the AWS outage that prompted this some time ago is
that, as it turns out, it wasn't hardware failure.
It wasn't malicious, it wasn't ransomware.
It wasn't anything silly like that.
Somebody made a mistake.
Literally a person made a mistake, and that caused a massive outage.
It can happen.
You and I are always the weakest link.
Humans are involved, and humans make mistakes.
But once again,
Amazon has the resources then to bring a lot more humans to bear on the problem
and get it resolved as quickly as possible.
And in fact, with respect to the Amazon outage, the article on which this video is
based includes a link to their after action report, their detailed report
of exactly what broke and what they did to fix it personally.
As someone who hosts, there, I truly appreciate that kind of transparency.
It's not knowing what they're doing that would make me uncomfortable
but by being upfront and acknowledging the error and detailing exactly not only
how it happened, but what they've done to not have it happen again.
That's great.
That makes me feel a whole lot better and happier.
Hosting, there hope that was interesting for updates for related links,
for comments and more, visit askleo.com/26591 26591.
I'm Leo Notenboom and this is askleo com. Thanks for watching.