- When I say we store and handle a lotta data
for a YouTube channel, I mean it.
I mean, we've built some'n sick,
hundred plus terabyte servers
for some of our fellow YouTubers,
but those are nothing
compared to the two plus petabytes of archival storage
that we currently have in production in our server room
that is storing all the footage
for every video we have ever made, at full quality.
For the uninitiated,
that is over 11,000, Warzone Installs worth of data.
But with great power comes great responsibility,
and we weren't responsible.
Despite our super dope hardware, we made a little oopsie
that resulted in us permanently losing data
that we don't have any backup for.
We still don't know how much,
but what we do know is what went wrong
and we've got a plan to recover what we can,
but it is going to take some work, and some money,
thanks to our sponsor, Hetzner.
Hetzner offers high-performance Cloud Servers
for an amazing price.
With their new US location in Ashburn, Virginia,
you can deploy cloud servers in four different locations
and benefit from features like, Load Balancers,
Block Storage and more.
Use code LTT22 at the link below for $20 off.
(upbeat music)
Let's start with a bit of background on our servers.
Our archival storage is composed
of two discrete GlusterFS clusters.
Both of them spread across two 45Drives Storinator servers,
each with 60 hard drives.
The original petabyte project,
is made up of the Delta 1 and Delta 2 servers,
and goes by the moniker Old Vault.
Petabyte project two, or the New Vault
is Delta 3 and Delta 4.
Now, because of the nature of our content,
most of our employees are pretty tech literate
with many of them even falling
into the tech wizard category.
So, we've always had substantially lower need
for tech support than the average company.
And as a result, we have never hired a full-time IT person,
despite the handful of times, perhaps including this one,
that it probably would have been helpful.
So, in the early days, I managed the infrastructure,
and since then I've had some help from both outside sources,
and other members of the writing team.
We all have different strengths,
but what we all have in common
is that we have other jobs to do,
meaning that it's never really been clear
who exactly is supposed to be accountable
when something slips through the cracks.
And unfortunately, while obvious issues
like, a replacement power cable
and a handful of failed drives over the years
were handled by Anthony,
we never really tasked anyone
with performing preventative maintenance
on our precious petabyte servers.
A quick point of clarification
before we get into the rest of this.
Nothing that happened as the result of anything
other than us messing up.
The hardware, both from 45Drives and from Seagate
who provided the bulk of what makes up
our petabyte project servers,
has performed beyond our expectations
and we would recommend checking out both of them,
if you or your business has serious data storage needs.
We're gonna have links to them down below.
But even the best hardware in the world can be let down
by misconfigured software.
And Jake, who tasked himself
with auditing our current infrastructure,
found just such a thing.
Everything was actually going pretty well.
He was setting up monitoring and alerts,
verifying that every machine would gracefully shut down
when the power goes out,
which happens a lot here for some reason,
but he eventually worked his way around
to the petabyte project servers
and checked the status of the ZFS pools or Zpools
on each of them.
And this is where the Kaka hit the fan.
Right off the bat, Delta 1 had two of its 60 drives faulted
in the same Vdev.
And you can think of a Vdev,
kind of like its own mini RAID array
within a larger pool of multiple RAID arrays.
So, in our configuration where we're running RAID-Z2,
if another disc out of our 15 drive Vdev
was to have any kind of problem,
we would incur irrecoverable data loss.
Upon further inspection, both of the drives
were completely dead,
which does happen with mechanical devices
and had dropped from the system.
So, we replaced them and let the array start rebuilding.
That's pretty scary, but not in and of itself a lost cause.
More on that later though.
Far scarier was when Delta 3,
which is part of the New Vault cluster
had five drives in a faulted state
with two of the Vdevs having two drives down.
That's very dangerous.
Interestingly, these drives weren't actually dead,
instead, they had just faulted
due to having too many errors.
So, read and writers like this
are usually caused by a faulty cable or a connection,
but they can also be the sign of a dying drive.
In our case, these errors probably cropped up
due to a sudden power loss
or due to naturally occurring bit rot,
as they were never configured
to shut down nicely while on backup power,
in the case of an outage.
And we've had quite a few of those over the years.
Now, storage systems are usually designed
to be able to recover from such an event,
especially ZFS, which is known for being
one of the most resilient ones out there.
After booting back up from a power loss,
ZFS pools and most other RAID or RAID like storage arrays,
should do something called a scrub or a re-sync,
which in the case of ZFS means
that every block of data gets checked
to ensure that there are no errors.
And if there are any errors,
these errors are automatically fixed
with the parity data that is stored in the array.
On most NAS operating systems,
like TrueNAS, Unraid or any pre-built NAS,
this process should just happen automatically.
And even if nothing goes wrong,
they should also run a scheduled scrub every month or so.
But our servers were set up by us a long time ago on CentOS
and never updated.
So, neither a scheduled nor a power on recovery scrub
was ever configured.
Meaning the only time data integrity
would have been checked on these arrays,
is when a block of data got read.
This function should theoretically protect against bit rot,
but since we have thousands of old videos,
of which a very, very small portion
ever actually gets accessed,
the rest were essentially left to slowly rot
and power lost themselves into an unrecoverable mess.
When we found the drive issues,
we weren't even aware of all this yet.
And even though the five drives weren't technically dead,
we erred on the side of caution
and started a replacement operation on all of them.
It was while we were rebuilding the array on Delta 3
with the new discs,
that we started to uncover the absolute mess of data errors.
ZFS has reported around 169 million errors
at the time of recording this.
And no, it's not nice.
In fact, there are so many errors on Delta 3
that with two faulted drives in both of the first Vdevs,
there is not enough parity data to fix the errors.
And this caused the array to offline itself
to protect against further degradation.
And unfortunately, much further along in the process,
the same thing happened on Delta 1.
That means that both the original and new petabyte projects,
Old and New Vault, have suffered nonrecoverable data loss.
So, now what do we do?
In regards to the corrupted and lost data, honestly nothing.
I mean, it's very likely
that even with 169 million data errors,
we still have virtually
all of the original bits in the right places.
But as far as we know,
there's no way to just tell ZFS,
"Yo dawg! Ignore those errors, you know,
"Pretend like they never happened,
"tow easy ZFS" or something.
Instead then, the plan is to build a new
properly configured 1.2 petabyte server,
featuring Seagate's shiny new 20 terabyte drives,
which we're really excited about like,
these things are almost as shiny
as our reflective hard drive shirt, lttstore.com.
And once that's complete,
we intend to move all of the data
from the New Vault cluster onto this New, New Vault.
- [Jake] All three.
- New New Vault.
Then we'll reset up New Vault,
ensure all the drives are good
and repeat the process to move Old Vault's data onto it.
Then we can reformat Old Vault, probably upgraded a bit
and use it for new data.
Maybe we'll rename it to New, New, New Vault.
Get subscribed, so, you don't miss any of that.
We'll hopefully be building that new server this week.
Now, if everything were set up properly
with regularly scheduled and post power loss scrubs,
this entire problem would probably have never happened.
And if we had a backup of that data,
we would be able to simply restore from that.
But here's the thing, backing up over a petabyte of data
is really expensive.
Either we would need to build a duplicate server array
to backup to, or we could back up to the cloud.
But even using the economical option, Backblaze B2,
it would cost us somewhere between
five and 10,000 US dollars per month,
to store that kind of data.
Now, if it was mission critical,
then by all means it should have been backed up
in both of those ways,
but having all of our archival footage
from day one of the channel
has always been a nice to have
and an excuse for us to explore really cool tech
that we otherwise wouldn't have any reason to play with.
I mean, it takes a little bit more effort
and it yields lower quality results,
but we have a backup of all of our old videos.
It's called downloading them off of YouTube
or Floatplane, if we wanted a higher quality copy.
So, the good news, is that our production 1X server
is running great.
With proper backups configured,
and this isn't gonna have any kind of lasting effect
on our business,
but I am still hopeful that
if all goes well with the recovery efforts,
we'll be able to get back the majority of the data,
mostly error free.
But only time will tell, a lot of time
because transferring all those petabytes of data
off of hard drives to other hard drives,
is gonna take weeks or even months.
So, let this be a lesson,
follow proper storage practices, have a backup
and probably hire someone to take care of your data
if you don't have the time.
Especially if you measure it in anything other
than tenths of terabytes,
or you might lose all of it.
But you won't lose our sponsor, Lambda.
Are you training deep learning models
for the next big breakthrough in artificial intelligence?
Then you should know about Lambda,
the deep learning company.
Founded by machine learning engineers,
Lambda builds GPU workstations, servers,
and cloud infrastructure for creating deep learning models.
They've helped all five of the big tech companies
and 47 of the top 50 research universities
accelerate their machine learning workflows.
Lambda's easy to use configurators let you spec out
exactly the hardware you need
from GPU laptops and workstations
all the way up to custom server clusters
and all Lambda machines come pre-installed
with Lambda Stack,
keeping your Linux machine learning environment up to date
and out of dependency hell.
And with Lambda Cloud,
you can spin up a virtual machine in minutes,
train models with 4 NVIDIA A6000s,
at just a fraction of the cost of the big cloud providers.
So, go to Lambdalabs.com/linus
to configure your own workstation
or try out Lambda Cloud today.
If you liked this video,
maybe check out the time I almost lost
all of our active projects when the OG 1X server failed.
That was a far more stressful situation.
I'm actually pretty relaxed right now
for someone with less much data on the line.
- [Jake] Yeah, must be nice.
- Yeah, I'm doing okay, thanks for asking.
I mean, I'd prefer to get it back, you know.(chuckles)