Hello and welcome to C++ Weekly. I'm your host Jason Turner, available for code reviews and
on-site training. In this episode I, well, I want to talk to you about a topic I've
mentioned briefly before but I think it's becoming increasingly important. You need
to make sure that you are actually paying attention to this - I do kind of feel like
in many cases it should almost be criminal if you don't do this with your C++ library.
[Music]
Now first of all, of course, I want to remind you to check out links to the swag store in the
description below, as well as links to my Best Practices book and my Puzzle books that I have
created; but I also want to just address briefly whether or not I am using hyperbole when I say
it should be criminal to not do this with your C++ library. This also applies to C, just for
the record, and probably Rust, and just about any other programming language, but I don't think this
is hyperbolic. I think that I'm actually pretty serious here and I will show you why hopefully
as we work our way through this episode. So now you've probably clued in at this point that I am
referring to fuzz testing. This is a technique for testing your software for vulnerabilities.
The general premise is that there's a tool that generates random input into your project,
and you execute this random input and check to see if you've got any sort of problems in your
code - looking at the Wikipedia article here, such as crashes, code assertions, memory leaks, etc. So
you have to be instrumenting your code, and we've already discussed the sanitizers, and that is a
key part of correctly using fuzz testers. I'm going to show you some very specific examples
in a moment, but the general premise, again, is that there's some sort of random string of input
(I don't know what it is) that is going into your program in some way,
something like this, and you have your if statements and your for loops,
etc that are being used to process this input in some way.
There's probably lots of code paths through your program that you have never specifically tested
for this random set of inputs, and the tester is going to instrument your code and try to find
inputs that find novel and unique code paths, so if it finds something that manages to execute the
for loop but not execute the if branch here, then it will search for things that do execute
the if and things that execute the else case, for example. My notes here are getting increasingly
messy because I'm getting carried away with the fact that I can now write on my screen while I'm
recording these episodes, so I apologize for that. This is not a new thing. You really do need to
note that like everything else in computer science we keep having this scenario of what is old is new
again. This was first named fuzz testing in the fall of 1988, which if my math serves me
is approximately 34 years ago, so I think at this point it makes sense to pull up a couple
of specific examples. Now as always I'm using Compiler Explorer, and there will be a link to
this specific example in the video description below, so again, make sure you check out all the
details there. What I have here is actually very representative of code that existed in ChiaScript,
my open source C++ scripting engine that I haven't maintained in the last several years, many years
ago when I first learned about fuzz testing. The premise here is that I have a parser of some
sort. I have a string_view input, and I check to see if the first character is an open brace,
and if it is then I continue to read through the input here, with this while statement, looking
for the close brace, at which point I return the index of this closed brace from the function. If I
don't find this matching pair of open and closed braces, then I return some sort of error value.
Now this is fairly poorly written - it's written for the sake of illustration, but again, it is
directly taken from an error that I used to have in my parser. If you've been staring at this code
you will have probably observed what the problem is at this point, and the problem is that I have
an unchecked loop. I am going to continuously go through this loop looking for a close brace,
and it will, if I don't have one, do a bunch of memory reads out in the middle of who knows
where. So I have set up this program so that it can be instrumented and tested with fuzz testing,
and that is specifically this LLVMFuzzerTestOneInput. This is an input function
for the fuzzing sanitizer specifically. It takes a blob of data and a size. Now this should be
fully defined behavior to do this reinterpret_cast to const char * because we are allowed to observe
any data through the lens of a character pointer, unsigned character pointer, or byte according
to the standard. Now if you can link me to a specific place in the standard where this code
is incorrect, please do that, but I do not believe that that is possible. So I take this data,
I do a reinterpret_cast to create a string_view,
and I send that string_view to the parse function. Now this little "." here
and this print statement are simply so that we can get an idea of how many iterations the program is
running through before it finds this bug in our code. Now keep in mind this code's not going to
generate any warnings or errors with clang-tidy or with any static analysis tool because it's,
well, not any that I know of, because they're just not smart enough to trace through this. Now
let's go ahead and take a quick look at how I've set this up. I have -Wpedantic, -Wall,
-Wextra, -Wconversion, -Wunused, -Wshadow. I have optimization enabled at the moment, all warnings
as errors, and then I have my sanitizer set up as fuzzer, address and undefined sanitizer. I had to
do this -fPIE to get it linking properly. So this is Compiler Explorer, and I have
it actually executing the code here, in case you're not aware that you can do that. We don't
care about the assembly output because it has all kinds of address and undefined behavior sanitizer
instrumentation that's been added for us. What we do care about is what the actual execution looks
like, and we can see that it started with this seed value because it's random, or pseudo-random,
inputs and we can see that we actually have an address sanitizer crash (heap buffer overflow)
come up quite quickly, and we can see that the crash happens on line 14, which is, well,
precisely where we expected to find this bug. That is right here. We can see where the crash
happened, that it is a heap a buffer overflow. It tells us where the memory was allocated,
which was allocated inside the address sanitizer, because we're using a string_view here,
and we can get the output here and we can see that it crashed on iteration 254 with
the simple string input of open brace. So all it took was that one case of an open brace
with nothing following it for it to get a crash, and interestingly we can take a little bit of a
deeper dive in here and just tell it to rerun this execution and see that we got a different seed.
We can see that it took 1964 iterations this time to find it, and it appeared to be the string of
an open brace followed by a new line. So there you have it. That is fuzz testing in a nutshell.
I have one other example that I would show you because I'm sure you're sitting here going,
"But my library doesn't parse any kind of user input." Well that shouldn't stop you.
Here is a relatively simple exercise. I just want a function that can do some sort of 32-bit rotate,
and I have a input value, and I have the number of bits by which I want to rotate this 32-bit input.
Again, I have this LLVMFuzzerTestOneInput. Now in this case I've created a handy little tool
for you that says make_fuzzed_objects, and I want to construct a uint32_t followed by an int taking
the input from the LLVMFuzzerTestOneInput, and then I use std::apply to apply this set of args,
which is the tuple of unit32_t followed by an int to the function called rot32 - kinda works
like that. I don't think we've ever actually shown std::apply on this particular channel. So
it executes as long as it can, and we can see that we have gotten an undefined behavior error because
the shift exponent 32 is too large for a 32-bit type, so we can see that this input function
is not safe within what is allowed for undefined behavior. Then we get another failure here where,
uh, the value 2570 is executed, and then we get a signed integer overflow in one particular case.
So it looks like we have lots of things that we need to actually test for this function,
and again, check out the video description for these links and you can get access to this link
that helps get you set up with this fuzzed object maker thingy. I'll have to be fair, it's kind of
slow to compile, but it makes use of many ellipses and lots of lambdas, which as you know, we love
around here. Okay so on the off-chance that you aren't convinced yet, I want to remind you of this
security vulnerability that was found in openssl where someone did a study here to say, "What would
have happened if I had run fuzz testing on openssl at the time?" At the time we're looking at only
six hours of fuzz test testing to have found this crash, and if you consider how critical
of a piece of infrastructure openssl is, then six hours really doesn't mean that much. Now I
would like to also point you to the project called OSS-Fuzz, which is a project from Google that does
continuous fuzzing of important pieces of open source software, and it fuzzes C, C++, Rust, Go,
Python and Java code using the similar techniques that we've already discussed here. As of June 2021
we can see that it has found over 30,000 bugs in 500 open source projects. You can see that openssl
is one of the projects that is being continuously fuzzed by this project. This is, again,
a critical piece of infrastructure. Now I want you to play a simple thought experiment with me.
What happens if someone discovers a flaw in libpng that could give an attacker root access
to your system in an unfortunate scenario, or in some way cause a crash, or a memory error
in your web browser, or whatever tool is using libpng. Now if that is found today it's probably
something that would become known as a zero-day exploit, because as soon as this is discovered,
approximately every device on the planet has some sort of vulnerability now because basically
everyone uses libpng. So no surprise, we can see that libpng is, in fact, one of the libraries
that is being continuously fuzzed by OSS-Fuzz, which brings me back to the beginning here.
It goes kind of like this - if you aren't fuzzing your own project, someone else is, and if you have
a project that is a key piece of infrastructure in your organization or in a public, open source
software, then arguably it could be nearly criminal negligence to not run fuzzing on
your library because you have almost certainly security vulnerabilities that someone else has
already discovered and has an exploit for. So perhaps I'm overstating this, but I don't think I
am. Leave a comment what you think, and please go out there and start fuzz testing your libraries.
[Music]