>>Matt Lowrie: So next up, we have Kostya, going to give a lightning talk.
>>Kostya Serebryany: Good afternoon.
I hope you're a little bit awake.
But to make you more awake, let me ask you a couple of questions.
First of all, how many of you care about C++ applications, stability and security?
Too little.
Everyone, please raise your hands.
[ Laughter ] Because you do care about stability and security
of C++ applications, even if you don't develop them yourselves.
How many of you have fuzzed something in the past, C++ or not?
Okay.
At least some of you in the audience know what fuzzing is.
I'll talk about fuzzing, and then I'll talk about one particular tool to do fuzzing.
And I'll have (indiscernible) live demo today.
So what is fuzzing?
Why do you want to fuzz?
What do you want to fuzz?
How do you want to fuzz?
A little bit about a particular fuzzing tool, and the live demo.
Let's go.
So what is fuzzing?
You can define it in many ways.
But, essentially, it is an infinite loop of create a test input for application and feed
it into your application.
There are two things you want to get out of fuzzing.
First, you want to crash your application.
So if you crash your application, you found a bug.
Even if you don't crash application, you still grow code coverage overall, and you can use
the -- the corpus you've grown for regular regression testing.
Why would you like -- why would you want to fuzz something?
First of all, C++ offers you many unique ways to (indiscernible) yourself into the feed,
like buffer overflows, user-after-free overflows and some such.
And fuzzing is a very efficient way to find such C or C++ bugs.
But even if you're not using C++, you're using some safer and slower language, like, I don't
know, any other, -- [ Laughter ]
-- you -- you may still have divide by zero, you may still have nulled reference, you may
still have assertion failures, you may have concurrency bugs, you may have bugs that cause
your application to consume too much resources, CPU memory, or network bandwidth or whatever.
Sometimes, actually, fuzzing may allow you to find logical bugs as well.
What to fuzz.
The usual suspect for fuzzing is any application or any library that consumes untrusted data.
So anything that consumes files from the Internet or from users, anything that monitors the
network activity, which accounts to more or less anything; right?
There are two specialized targets, operation system kernels and UI applications.
And I'm not touching those two.
How to fuzz.
One of the very popular ways to fuzz is to apply some kind of application-specific generation-based
rule.
For example, if you know the syntax of the files that your application consumes, you
can generate files that follow this syntax.
And this approach has been very, very useful in many cases at Google and outside.
But I'm not touching this today.
Another approach is called mutation-based fuzzing.
This is when you already have a set of valid inputs or invalid inputs for your application
and you start mutating random inputs in random ways and feeding these inputs into your application.
And sometimes mutations could be as simple as bit flipping.
Sometimes mutations are much more complex.
And this approach is also very powerful.
But if we make a little -- one tiny step to improve it, it will become much more efficient,
like orders of magnitude.
And we've seen it in many cases.
So the tiny step is called guided mutation fuzzing.
And by guided, I mean that we use code coverage to guide both the mutation and the extension
of the corpus.
The work flow is as follows: You build your application with some kind of coverage instrumentation.
So every time you execute an input with the application, you know what parts of the application
have been touched.
And then you start mutation fuzzing.
If some of the mutations caused new code coverage to appear, you say, oh, this is a good input;
it touched part of application that the rest of the test corpus didn't.
So we added back to the corpus.
And we have lots of evidence that this increases the efficiency of fuzzing by several orders
of magnitude.
Come on.
Yep.
We are working -- "we," meaning my team right now -- are working on a very specific way
of fuzzing.
We fuzz C and C++ APIs or libraries.
We do not right now fuzz large binaries.
We do not fuzz services.
We fuzz small or medium-sized API entry points.
You can think of it as unit test-like fuzzing.
So we define a thing which we call fuzz target, which is a C function -- a C function that
consumes bytes.
And inside that function, you can do whatever you like with these bytes, basically, feed
these bytes in one way or another into your library.
Who can find the bug on this slide?
>>> (Off mic.)
>>Kostya Serebryany: Yes, correct.
So on this slide, you have C code that, given one specific input, will trigger buffer overflow
on the line which says data sub 3 equals Z.
Can everyone see the bug?
I leave it here for ten seconds.
Raise your hand when you see the bug.
Come on.
So fuzzer will find this bug much quicker than you.
[ Laughter ] There are lots of -- there are several fuzzing
engines on the market.
And the fuzzing engine is the tool that finds inputs that are interesting enough and feeds
them into your -- into your fuzz target, into your application.
One of them is developed by my team.
It's called libFuzzer.
It's an engine for guided in-process fuzzing.
Guided mean it is using code coverage.
And in-process means that everything it does, it does in a single process.
It does the mutations and it feeds the inputs into the API and the test all in the same
process.
It's a library in a regular C++ sense, which provides a main function.
If you want to use this library, you need to build your code with some extra compiler
flags that actually do provide the code coverage information.
Then you link your application with the libFuzzer.
Then you go in parallel -- or before that, you have to find a set of inputs for your
application somewhere, an initial test corpus.
Sometimes it's okay to have initial test corpus empty.
But in most cases, the better test corpus you have, the more efficient fuzzing will
be.
And once you have this directory, you just pass this directory as a parameter to the
-- to the just-compiled binary of the fuzzer.
This is it.
Let me show you some example.
This bug was made public about three or four -- four weeks ago.
It was a buffer overflow in one of the network -- one of the open source network libraries,
CA Res, and you have the CV, and the name's on the screen.
The Bug in that library was a single byte buffer overflow which overwrote one byte somewhere
outside of the object.
It caused a full exploit of Chrome OS.
So there was direct impact for Google, that Google had to pay $100,000 for the researcher
who provided us the information, but there was also indirect harm, because, who knows?
Maybe some -- someone else has discovered this bug before and was using it against our
customers.
So this is a pretty nasty -- this was a pretty nasty bug which is now fixed, and all you
need to find this bug is on this slide.
Everything you need is on this slide.
So count how many lines here.
Can we switch
to demo?
Okay.
So I have -- I have a pre-built library binary of the fuzzer
that I've just shown you on the screen.
How many of you believe me that I can find a $100,000 worth bug before the end of this
talk?
Come on.
Okay.
Done.
So I've started the fuzzer on a single CPU on this laptop, and this is not the most recent
laptop, and it started finding new inputs.
And on the very left you may see the numbers.
The most significant digit is eaten, but it basically executed several thousands of inputs
before it found an input that triggers a bug.
And the bug is heap buffer overflow, right of one byte.
And here is where.
So if someone invested ten lines of code into finding that bug, he would find it instantly,
as you have seen.
What else is convenient about libFuzzer is that once it reports a bug, it also creates
a found disk which has the complete -- which has the bytes that caused the bug to happen.
And if you want to reproduce it -- can you see it?
Yeah.
If you want to reproduce the bug, all you need is to feed these bytes again into your
target function like this.
And it will happen again.
And so you can debug it.
You can verify that it's fixed.
You can add this input to your regression, test suite, et cetera.
Switch to the slides, please.
So I've just shown you a very, very important, very scary bug that can be find by fuzzing
instantly.
Fortunately or not, not all scary bugs are like this.
For some bugs we need to spend hours.
For some bugs we need to spend months finding them.
And Google has these resources and Google has, to some extent, humans to do this.
Not all open source projects have it.
And we're trying to help.
So what do we want to do?
We want every Open Source Project, at least every important, every significant Open Source
Project to move away from it has been fuzzed at some point or it has never been fuzzed
to the situation where we can say this project is being continuously fuzzed, and we actually
know that the coverage produced by the fuzzing is good enough.
And for that we are starting to introduce what we'll call Oss-fuzz, open source fuzzing,
which is a fuzzing as a service for important open source projects.
It is based on the same fuzzing infrastructure that we're using for Chrome, and that has
found many thousands of bugs for Chromium so far.
It provides multiple fuzzing engines under the hood, but the user shouldn't know about
it.
All the user should know is how to interface -- how to create the glue between their project
and the OSS fuzz.
And this slide again is an example of such glue.
The project is now public.
It's on github.
And if you own or work with some Open Source Project, you're welcome to participate.
Thank you.
And I have 55 seconds for questions.
[ Laughter ] [ Applause ]
>>Matt Lowrie: Excellent talk.
Any questions from the audience?
We have one.
>>> So in terms of, like, detecting, like, memory issues, like, how does this compare
against, like, Valgrind and stuff?
Like how much effort and time do I have to spend with this tool versus something like
Valgrind which has been out there forever?
>>Kostya Serebryany: Forget about Valgrind.
Use Address Sanitizer.
But these -- Valgrind and Address Sanitizer works as fuzzers, are complementary things.
You cannot find bugs with Valgrind and Address Sanitizer just by running tests because the
tests will not execute the buggy code.
You need the fuzzer and the tool like Valgrind, Address Sanitizer together to interesting
find bugs.
>>> My question is how is the actual mutation done?
Can you provide some details on that?
>>Kostya Serebryany: We start from bit flipping and byte flipping, those simple things.
But also, as fuzzing runs, we gather the statistics about the comparison instructions and library
calls, and we actually accumulate a dictionary of tokens that are used by the input language.
And then we start feeding these tokens into the input stream.
So over time, it gets much more sophisticated than byte flipping and bit flipping.
>>Matt Lowrie: I think that's -- all we have time for for lightning talks, but thank you,
Kostya.