>> VAAGN TOUKHARIAN: Good day.
My name is Vaagn.
>> TIGRAN GEVORGYAN: And my name is Tigran.
>> VAAGN TOUKHARIAN: We will talk about HTTP time bandit but before we start our talk,
I have one thing to ask.
This is our first talk at DEF CON.
So don't be nice to us.
Be very, very, very gentle to us.
Please.
Thanks.
(Applause).
So about us.
Tigran and I work for Qualys.
We break stuff for fun.
Sometimes we fix them too.
We are interested in time travel and we'll talk about that a bit later.
And we both love to tri, we do triathlons.
So what we going to talk about?
The presentation short, but effective maybe.
You will judge by leaving the room or not.
It's going to be about another application, DOS method and we'll then show you the tool.
We'll go through some stats and we'll talk about what are good or bad uses of tool or
method and then we'll glance over the defense against this method.
Why?
Why do we do this?
It's obvious because we wanted to have time machine.
We couldn't buy any on eBay.
We could, but it was similar to what you see on the screen.
And that really doesn't work.
So we thought we will make our own time machine and I think we succeeded.
The time machine that we have, does one thing and does it well.
It can give you the future state of your web server given the resources you are going to
push on it.
About it a bit later.
So we going to talk about application layer, DOS tool or method.
For us, DOS tool are two different kinds.
There are some that are not that smart and another group that is a bit smarter.
Not smart apart is pushing your resource over and over and over and hoping the resource
will die.
And then you do that from multiple instances, it works.
But that doesn't have feedback.
And sometimes a load is near symmetrical.
Smarter bots like slow loris and slow HTTP test, and PKI abuse, they are smarter and
they are asymmetrical in nature.
Our proposed method is going to be striving to be asymmetrical.
>> TIGRAN GEVORGYAN: So how do we do this?
So the point of this talk, the information from future is picking the resources out of
all the available ones, picking the ones that are most painful for the server.
While people say, oh, servers done feel pain, we all know they do.
So how do we do this?
We spider the website, we collect, compile a list of the resources and then we start
measuring certain things, for example, the download speed or the start to download time.
What this gives us is a lot of data and what deals very well with a lot of data and that's
statistics.
So we use statistics to filter through the data.
We calculate the average of all the download times.
That gives us a baseline, what is a normal transfer time.
And then we can tell ‑‑ oh, these resources are slower or faster.
So we use to formalize this, we use the ‑‑ we used mean as the measure of central tendency,
and we calculate this standard deviation.
The resources that have high standard deviation, that means that they are inaccurate in terms
of they can be fast or they can be slow, but we can't really tell.
So we throw them away.
We select resources that have slower download times or start to download times and we think
that this is related to CPU load, and the reasoning is that all the transfer delays
are the same for all the resources, the network‑related ones.
So if some resource has higher transfer time, that must be resource specific and that makes
it interesting.
So this, for example, is a chart of resource download for a website and you can see that
the majority of resources fall close together in some middle range.
There are some outliers, so fast outliers and some slow ones.
The ones that are interesting are the slow ones.
The question is.
What causes these resources to be slow?
Is it the CPU load or is it just the sleep standard on the page.
So small demonstration and not to anger the demo Gods, we did a video.
(Laughter).
So here the tool is running against a locally hosted web server and when it's done it, generates
a list of URLs with the measurement mean, the standard deviation.
While this is nice, it's not very, very useful.
Instead, the tool can be constructed to output this information in the HTML.
We see it in increasing order.
So the HTML resources down at the bottom are the fast and the PHP resources are near the
top.
And this is because PHP requires the PHP interpreter invocation, which takes some additional time
and the resource at the top are some artificially made pie calculation, which understandably
takes a long time.
And so XML is very nice because it can be head into some other tools and I have one
example which is an XXLT which transforms this information into something that can be
visualized.
So you will see a scatter chart of download speed and standard deviation and there are
four quadrants here.
The top quadrant, the top two quadrants, they are the resources that have high standard
deviation, which means that the measurements were inaccurate.
So since we cannot trust these resources we throw them away.
Instead we concentrate on the resources on the bottom half.
On the right side, these resources are fast resources, and consistently so.
So they are not bottlenecks.
Instead, we concentrate on the resources on the lower left quadrant, and closer to the
zero zero point, the better ‑‑ I mean, the painful the resource would be for the
server.
So the tool also can be ‑‑ can generate data for visualization in other means.
For example, the next one, whenever it comes, it will be ‑‑ the tool is instructed
to generate graph dot language graph.
So here you can see a small website with transitions and each transition is marked with a transfer
time.
So slowly the resources could be highlighted.
I couldn't do a good job here, but the capability is there.
So next, the tool is going to scan real website, built with concrete CMS, and so we give it
the URL to start with, how deep into the website to go into and how many measurements for each
resource to take and write it into concrete.xml which will be processed later.
So once it's done, it's crawling right now.
Once it's done, it will generate the XML and then what can be done with this XML is, again,
feed back into the tool, which will correlate and find out which resources are slow in its
opinion, and it picks some PHP resources out of that.
And so it will start sending paralyzed requests to these resources and measure whether some
service degradation happened.
It will output, what was the original measurements and what were the new ones.
So by looking at these deltas, you can figure out whether this is truly CPU heavy resource
or not.
So now a bit about asynchronous nature.
So this slide shows three screens, the top one is the top utility output of the web server.
The bottom right one is the client.
You notice the client has 9, 9.5% usage, and that's mostly because of the recorder.
I'm launching the tool five times to why the most heavy resources and the load on the client
side, it was not ‑‑ it goes up to 9.5%.
So negligible, but the load on the server side goes up to 80 to 82%, which is, you know,
asynchronous in my book.
Okay.
So this concludes the demo.
>> VAAGN TOUKHARIAN: And now that we saw the tool, let's talk about what can be done with
it.
You can use it for the good of the humanity.
As a QA tool, for example, or to figure out if you are vulnerable to attacks described
a minute ago by just going in and taking all of those CPU hogs and debugging them or getting
rid of them, rewriting them.
Or you can use it for bad which is almost what Tigran showed two minutes ago, where
he paralyzes the resources and tries to see if the degradation happens, imagining that
being all automated and one click, boom, you go find the worst resources and then do more
analysis by figuring out which ones are the worst in real life, which ones degrade, and
then you push them even more.
And then, of course, there is the ugly, which is the bad, but distributed basically.
So that is what we have right now and what we were thinking of on adding on the tool
or a method is a couple of tweaks.
One would be the measurement of degradation, more precise measurement of degradation, in
parallel testing, which will go to nuances like understanding load balancers.
We don't know how load balancers would behave right now, how our calculations will be affected
by load balancers.
Those things need to be worked out.
Then we have a couple of other things that would be nice to add, like SQL wildcard injection
into forms that we think are search forms, or in any way are fit to SQL query.
And then another thing would be figuring out if state reset is pricey, to see if when you
are forcing an application to generate new tokens for session ID or whatever, it takes
any CPU load.
That has to be calculated, estimated, or guessed, or doctored.
And now let's talk about defense.
Well, what can be done on the defense side?
Load balancers are probably good measure here.
Our guess would be mod security here, and you can use the tool to identify and fix memory,
DBCPU hogs make your application work smoother and be less vulnerable to this attack.
And slide deck has a reference number one, which has a link to very simple mod security
setup that most probably going to cover 90% of cases.
It's done by some other guys but we find it's effective.
And then there is one more thing that Tigran thinks can be done.
>> TIGRAN GEVORGYAN: So I'm lazy, right?
If I can off‑load something to the computer, I prefer that.
So the idea here is to design a mode security extension that will keep track of the website
usage, meaning it will keep track of what resources ‑‑ the chain of resources that
was requested, the response times that were witnessed and all that, and it will track
and tag out of place requests.
For example, if a resource is requested all the time through some chain of events, and
this resource is requested immediately, that would, you know, warrant some special consideration.
Same goes for state coherent checks where a resource is always requested with some state
and now the request comes in without a state.
That's an indication to pay some close attention to that as well.
Yeah.
With that, we thank you.
>> VAAGN TOUKHARIAN: We are on time.
Thank you very much.
(Applause) Thanks to Sean for setting up the lab for
us.