(electronic music)
- Hi, my name is Patrice Godefroid.
I'm a researcher at Microsoft Research.
My area of research is software analysis,
testing, and verification.
Today I want to tell you
about some recent work we've been doing
on REST API fuzzing.
So what is REST API fuzzing?
Well, as you know, Cloud computing is currently exploding.
There are more and more Cloud services available every day.
Most Cloud services access programmatically
through REST APIs, both by other services
and by third-party applications.
REST APIs are implemented on top of the HTTPS protocol stack
and offer a uniform way to manage Cloud services.
Cloud service developers can document their REST APIs
using interface description languages like OpenAPI,
also known as Swagger.
An OpenAPI Specification describes
how to access a Cloud service through its REST API,
including what requests the service can handle,
what responses may be received,
and the exact formats for requests and responses.
The REST APIs can be very complex.
For instance, REST APIs of Azure services
are describing millions of lines of OpenAPI code
publicly available on GitHub.
To master this complexity,
we need new tools to prevent expensive bugs
in Cloud services.
At Microsoft Research, we have recently developed RESTler,
the first stateful REST API fuzzer.
RESTler is a tool that takes as input
an OpenAPI specification,
and then automatically generates an execute test
from that specification.
So, with RESTler, you do not need a test read
or prerecorded API traffic to start.
RESTler automatically generates API traffic from scratch
directly from an OpenAPI specification.
It will generate lots of requests,
some well-formed requests and some badly formed requests,
to try to trigger bugs
in the Cloud services being tested.
That's what fuzzing means.
During its testing,
RESTler can find all kinds of bugs in Cloud services,
such as HTTP 500 internal server errors,
which are unhandled exceptions.
That is, these are crashes
in the Cloud services being tested.
RESTler also checks for specific API properties
such as a used after free should never happen.
If RESTler creates a new resource and then deletes it,
RESTler should not be able to re-access
this specific resource later on after it'd been deleted.
If this ever happens, RESTler reports an error.
RESTler also includes checkers
for several other properties that should hold for every API.
So how does RESTler work?
Well, my colleague Marina will tell you more about this.
- Thank you, Patrice.
Hello everyone. My name is Marina Polishchuk.
I'm a software engineer at Microsoft Research.
RESTler has two main components: a compiler, and an engine.
So, the compiler analyzes a Swagger specification
and generates a fuzzing grammar,
which contains information about all of the schemas
of the requests and the responses
as well as producer- consumer dependencies.
Producer-consumer dependencies
are a unique feature of RESTler
that allow it to more deeply cover services
out of the box automatically.
What are producer consumer dependencies?
Well, if say one request produces or creates,
for example, a resource that is then consumed
or taken as parameters by subsequent requests,
RESTler will be able to infer that ordering constraint
and will be able to extract those ID's names
or some other property out of the responses
of the first request and specify in the specification
that the second request should take those as a parameter.
The RESTler engine will take the fuzzing grammar
and execute tests
and actually report bugs
that are found in the course of test execution.
Tests here are going to be sequences of requests,
and the engine uses a systematic state space search
to cover those sequences.
It also has some algorithms
to eliminate redundant requests
to avoid fuzzing the same things over and over.
So those of you who have done some fuzzing,
you may know that there's a lot of repetitive kind of coverage
that gets generated during fuzzing.
And RESTler, for example, will detect that
if say request ABC were executed in a sequence BAC
and C returned a 400 or bad request,
RESTler will say, "Oh, this means that
the service when passed an invalid input after that sequence
handles it gracefully,
and it doesn't have any bugs to report,"
so it'll avoid trying that sequence in the future.
There are four main modes in which RESTler works.
The first mode, compile, takes your API specification
and produces the fuzzing grammar
as well as some default configuration files
that you can then customize
such as a fuzzing dictionary and the engine settings.
The next phase is a test face.
The test face tries to execute each request
in the specification
and in as minimum number of steps as possible.
So, it will only execute the dependent requests
as well as the request itself.
And it'll try as many parameters as needed
to try to get a successful request to execute.
Then what it outputs is a coverage file,
and this file will tell you how many requests were executed
and how many of them were successful
and in detail, all of the other return codes
and responses for all the failing requests.
So, for example, if your service has 110 requests
and RESTler successfully executed 50 of them,
well, that might seem like low coverage,
but you may still get a lot of value
out of fuzzing those 50 requests
and so you may say you're not going to customize anything else.
You just go on and do fuzzing in the next phase.
Or in some cases zero requests may be covered.
The most common of those cases is
if there is some path parameter
that's really a magic value, it's not produced by anything
in this particular API,
for example, a tenant ID
that always has to be specified to this API,
then you can put that in the dictionary file
and try the test phase again
until you do get some positive coverage.
The next phase of RESTler is a fuzzing mode,
and we have several. We have a full fuzzing mode
as well as kind of a lean fuzzing mode
that allows you to quickly find some bugs
by executing the RESTler checkers,
which checks security properties
together with more efficient fuzzing algorithms.
The replay mode is the last mode of RESTler
and this allows you
to quickly reproduce all of the bugs that were reported
during fuzzing using RESTler itself
and the files that it produces.
We should mention that RESTler has been used at Microsoft
to find bugs in many services across office 365,
Bing, and Azure.
All of these bugs have been fixed.
You can find specific data if you're interested
in case studies in some of our recent papers,
such as the ICST 2020 and FSE 2020 conference papers.
Now, after you've tried RESTler and found a few bugs,
you may ask, "well, how do I run RESTler regularly?
What do I need to do to integrate RESTler
in my CSCD pipeline?"
Microsoft Research has also released
a REST API Fuzz testing toolkit,
which is a self-hosted Azure service that you can deploy,
and it has several tools configured out of the box,
such as RESTler and OS Zap, and a few more.
You can very easily add other API testing,
API fuzzing tools that you would like into this framework.
It'll also allow you to deploy your service under test
if it runs in a Docker container.
Now, you can use this,
for example, to run several tools in parallel,
if you want to scale your testing
to many check-ins across many tools,
or you can also perform differential regression testing
across different API versions or across time
for your services as they're being deployed.
- Thank you Marina.
For more information, please visit these links.
RESTler is available on GitHub,
where you can find in there our repo documentations
and examples and the code
and the research papers that we have published about it.
RAFT is also available in its own GitHub repo.
Please give these a try.
Thank you very much for your attention.
(electronic music)