>>Matt Lowrie: So next up we have a great way to finish the conference, a talk on testing
-- crashing servers by Jonathan Abrahams.
So I will hand it off to him.
>>Jonathan Abrahams: Get the slides up.
Hi.
I'm Jonathan Abrahams.
I work at MongoDB.
I'm in the test infrastructure group.
Hopefully some of you are familiar with what MongoDB does, and we're going to talk a little
bit about how we test our server under crash scenarios.
So as you probably are aware, machines can crash for a variety of reasons.
Maybe it's in a virtual machine or it might actually be a physical machine.
And we need to see what's going to happen to our MongoDB server.
It's pretty important.
We're holding data that's stored out to disk and we really don't know when we came to this
project, we weren't really sure can we be robust enough to survive that.
So there could be operating system failures, hardware failures, and then there's also application
crashes of the MongoDB itself where maybe it got some sort of unexpected termination,
maybe from fuzzing.
Well, we could actually try just aborting the mongod, which is the process of the server,
but it really doesn't fully simulate what happens if you have an unexpected crash of
the machine or, like, the operating system kernel.
We really want to be able to test some sort of immediate power loss because of caching
that might be done at the lowest level of the I/O, and also kernel panics can leave
the application and data in unrecoverable states.
So this is your typical startup of your machine after it's been terminated or even booted.
It will go into some sort of hardware and software checks in the operating system.
And then eventually it will turn it over to your application.
In our case, the mongod has its own recovery mode.
It can defect the fact that it had been terminated.
And if it passes that then it allows it to be connected to my clients.
So, you know, how are you going to go about -- when we jumped into this project, what
are you going to do about testing something like, you know, a power failure?
Well, the easiest thing is just basically run mongod on some machine in a lab or at
your desk and pull the plug and start it up again and see what happens.
Of course, that's manual and labor intensive and, you know, you can't do this across, you
know, different testing cycles.
So then the next -- the next thought was, well, we could do a real low-tech cheap solution.
Let's just put an appliance timer there.
You don't have to be there to actually pull the plug but the appliance timer will do it.
But the appliance timer is not fine grained.
It's 15 minutes.
You have to wait before you can actually turn it back on.
So you can't do these repeatedly over and over again and get maybe quick results.
Another alternative approach, which we'll discuss a little bit later how this was done,
was to use an internal command to actually crash the machine, which is almost functionally
equivalent to a power cycle.
And then I was able to set this up in a Cron job so we can run this internal command so
everything is kind of like running on that machine.
So that was another approach.
So we realized we really need to do this a lot more often, and, you know, so we have
to go figure out how are we going to be able to do this over and over again.
Because the recovery -- the moment at which you crash, it's very subtle.
We really don't know at one point what might be running that might cause something to be
unrecoverable.
So we want to run it like maybe, I don't know, a hundred times or something before we feel
good about this.
So we came across this solution, which I don't know if you can actually see it here.
This is what we use for the power part.
It's called the Ubiquiti End Power Pro.
Let me see if I can turn on the light.
And it's a programmable power device where you have SSH access to it, and all you do
is you can go in there and you can basically turn off one of the outlets on here or any
of the outlets with a very simple command.
This is how it's done.
Hopefully you can see that on the slides.
All you do is you send the zero to shut it off to some slash dev slash, let's say, output
eight which this is plugged into, and to turn it back on you send a one in there.
And that's all that script is doing right now.
So that was one part of the solution.
We figured out how we can crash physical machines and repeatedly turn them off and on.
What about virtual machines?
Virtual machines operate differently.
They use shared hardware -- or rather, they use shared resources because they rely on
the underlying host operating system.
And it's also important to test virtual machines for maybe terminations of the virtual machine
because that's what a lot of our customers use out in the cloud or even in their own
data centers.
They're actually running virtual machines.
So it's important to know from a completeness perspective whether or not virtual machines
are going to have any problems as well as physical machines.
Crashing virtual machines actually turned out to be fairly simple.
You know, like using KVM type of machines using the Linux containers.
It's just using this virsh destroy command which is -- you know, there's nothing sent
to the underlying applications running in the machine.
It's like basically destroying it, and -- but it doesn't destroy the machine from the perspective
you can't bring it back up.
It just basically -- it's like a shutoff.
And VMWare also has this VM run stop hard command.
And turning it back on is the same corollary.
You're just running start commands.
And believe it or not they actually do come back on.
I did script this, and we used both of these, and you could probably do this for other virtualization
-- you know, other virtualization machines as well.
So here's the other part that I did uncover about internal crashing which works for Linux
only.
There's this thing called a sysrq key which is like as if you're at the console and you
want to basically put it into some boot mode, you can do this through software by writing
a 1 out to a proc sys kernel sysrq, sets it up, and then you send a B, for boot, to proc
sysrq trigger.
It's an immediate boot of your machine.
So here's another part of where you can basically internally take any Linux machine, I think,
you know, most modern ones, at least, and it will just boot it right up.
So it's like another type of crash.
One other part that we stumbled upon was that when the machines do come back on and they're
plugged into something, you have to make sure your BIOS is set so that when it sees the
power, it will bring it back up.
So that's another one of those caveats.
Windows machines have some sort of additional prompting that will happen sometimes, so that
was another thing I was able to script in.
So I was able to actually do power cycles on Windows machines but through the physical
machines.
And there's a utility called BCD edit, and you can set the boot status policy to ignore
those failures so it doesn't give you these prompts like do you really, you know, want
to go into recovery mode?
Do you want to boot normally?
So essentially you can just basically plug a machine into here, run some sort of script
that crashes it, and whether it's plugged into a Linux machine or Windows machine, they
will come back up.
If you want to run an application, in our case mongod, you can go ahead and run your
applications as you would normally and then, you know, do whatever tests you want to do.
So we had the machine -- we figured out all these key parts to it.
We stitched them together into a batch script.
And, like I said, before, the mongod goes into this recovery mode.
This is performed automatically.
In the case of our storage engine WiredTiger, which is what I have on my shirt which is
our default storage engine now, it's going to go and look for some stable -- the last
stable copy of the data it has on disk from the last check point it wrote out and then
it will apply a journal log to bring it back up.
It's a little bit more we want to do in the scripting.
Before we actually crash a machine, we actually want to apply some sort of moderate load or
maybe even more than moderate of all sorts of different type of database commands, and
to basically put the database server into any mode we can think of so that at the time
the crash happens, we'll be able to, you know, possibly stimulate a data corruption or something
like that.
That's what would be our failure.
And in our case, you know, that was specific to the MongoD server.
We added in some other options about, well, how we wanted the clients to do that, whether
that would be in a replica set, a single-node replicate set, or stand-alone, what storage
engine to test.
When we started the testing, WiredTiger was new to MongoDB.
We actually had just acquired it, and it was being introduced as a second storage engine.
We used Nmap v1 as our storage engine prior to 3.0.
And there was a lot of focus on making sure that WiredTiger, you know, was really going
to be robust enough to switch it.
So after it started, like I said, we're going to do some other things, check the status
of the server.
But the key point is that if we fail to recover or we can't connect to the MongoD because
maybe it died or something like that, and some other validation that we can run against
the MongoD once it comes back up, we know the test has failed.
Okay.
And this basically talks a little about the type of things that we do after we've restarted
it.
The validation is an additional step.
For those of you who do use MongoD, that does not happen on recovery mode.
Recovery mode is not as -- it doesn't look through all the documents to make sure that
it's -- there's no problems.
But validation actually goes through your collections thoroughly.
It's a slow process.
But it might be necessary to do if you do encounter this type of situation.
What was good about this project is, we developed a script to make it basically like a self-service
application, because we had developers all over the world.
And this script actually can -- not only can run on -- it doesn't need any customer hardware.
It can run on Windows machines, it can run on Linux machines, it can run in the cloud,
it can run on localized and remote hardware, things like that.
And we'll save some data for -- in case of failure is detected, we'll back up data for
our developers to use.
So, in conclusion, we found that the crash testing helped to extend our testing to scenarios
we had not previously covered.
And to be able to provide these local and remote teams with the tools to reproduce and
analyze the failures.
And we hope to improve the storage layer hopefully for the MongoDB.
Like I mentioned before, we did find some bugs there.
And here's a list of some of the bugs that we found.
Most of them are fixed.
I think there might be one that they're still working on.
One issue that I did find that we haven't been able to solve, maybe we have somebody
out here on the Windows crowd.
You can easily crash, like I said, internally, Linux machines.
But from Windows, you get these blue screens of death all the time, but there's -- I don't
believe there's any systematic way to crash a Windows machine.
If anybody knows, you know, please send me an email or be in touch with me afterwards,
and I'd love to hear about that.
Thank you.
[ Applause ] >>Matt Lowrie: Thanks, Jonathan.
So tools to crash computers.
I am picking up so many great things from this conference.
>>Jonathan Abrahams: You can turn my light back on.
>>Matt Lowrie: So we do have a question from the audience.
How do you verify that all committed changes are present after the recovery?
>>Jonathan Abrahams: All committed changes are present after the recovery?
Oh, okay.
So that -- yeah, that's a good question.
So that's part of the validation.
Right.
So one of the things that -- one of the options that I mentioned before was, I write a check
point document now before I do the crash, which is an optional thing, because also that
can trigger certain things that can happen at the database layer.
So if that doesn't show up, then, you know, we have what we call an (indiscernible) and
a replica set, an op log.
So, if that shows up, then we should have everything prior to that.
So that's one of the ways we're able to determine that.
But, you know, if your application -- you know, you want to do something like this,
you probably have to come up with some ways of figuring out how you're going to be able
to determine that it's successful also from recovery.
>>Matt Lowrie: All right.
Any questions in the room?
Well, then, thank you, Jonathan.
>>Jonathan Abrahams: Okay.
[ Applause ]