[intro music]
[theme music]
Hi. Today we're going to talk about
not fixing the symptoms of a problem.
and actually learning to do classic real-world performance.
And within the Real World Performance group, we always take pride
in taking a top-down view of the problem, rather than debugging a problem from the
bottom up.
And this is a big challenge that we face in our industry, particularly in IT
departments that are heavily segmented
and very often don't communicate well with other departments.
And everybody's working defensively to make a show that everything looks good
from that perspective.
And people aren't learning to look holistically across the whole
depth of a system to see how we can increase the whole system performance.
And this is a big challenge within today's
IT organizations. And it's something that I take great pride in that the Real World
Performance group does is that we look holistically
over the whole stack and see what can be done
to improve the whole system performance. And
this is extremely important and what we have is a very simple
demo that we're going to show and it probably is being panned out
and played out the similar situations in multiple companies,
in multiple industries, in similar applications
all around the globe. And people are scratching their head
thinking "How do we increase performance?" And a lot of people are making lots of
suggestions,
but they haven't yet figured out what the problem is. And
this is something we see daily
with this desire to be fiddling with things, changing things.
But really no sort of understanding of what the big picture problem is.
So on the system what we're looking at today,
it actually record... it's coming...it's processing records coming off from
manufacturing facility.
Basically bar codes on devices are going through on testing equipment and
manufacturing equipment,
and it's generating a log. And it's producing
manufactured goods. And all those manufactured
logs and things like that are going into flat files. And then we have to load
those files into a manufacturing database
such that we can figure out how efficient, ah, is the production line running?
What's our defect rates?
Is there a genealogy associated with bad machines which have a high error rate? And
things like that.
But this could be the same as many other applications. This could be your
telephone bill.
This could be an online application where all requests are coming from a
single
browser. It likewise could be from a government agency processing tax returns
or tax forms and things like that.
There's so many times we have seen this
occur, um, within the real world
yet we have seen a lot a very smart people
focusing their effort literally trying to do bottom-up debugging
fiddling around with database parameters and actually having
absolutely zero success of raising
the real world performance number, which is the business transactions, that they
push through this system.
And so this is just, a hopefully a
a bit of a wake-up call, and an education. When we actually built this demo, we were
surprised
at the performance gains that we can achieve
by just doing the basics of good computer science
and doing a top-down analysis. So maybe I should introduce the system.
It's a workload that is loading manufacturing data
from a series a flat files. And,
the application is basically reading the flat files
and then basically hashing the data of each row
to a number of application servers, and it's doing this as a means of load balancing
everything.
All the work that happens in all the rows that arrive are load balanced across the
application servers,
which is what you can see with the red lines there. And then the application
servers load balance across
to two instances within an Oracle database. Now let's have a look at the
performance of the system
and by the way this is not good performance. This is our start. This is
our entry point
at the start of the analysis. You'll notice the transaction rate, um,
is about 8,000 transactions a second, maybe
closer to 7. You notice we're spending a lot of time waiting
in the app server. The database response time is getting close to 40 milliseconds.
And if we actually start looking at the ASH report or the AWR report, you can
see
that we're actually not spending much time at all
on the CPU, which would be that thin green line at the bottom,
bearing in mind we've got two instances. But boy! Are we
spending a lot of time in sort of rack
related move cache fusion events moving blocks
back and forward between database instances.
So at this point in time people are thinking, well
"What's going on here? The databases doesn't scale."
So perhaps we can just step through
and just start to understand the problem. People are saying, um,
you know, perhaps we need to rebuild the indexes, hash partitioned,
reverse key indexes. People are saying build or rebuild indexes with higher
percentage freeze,
reset INI_TRANS to a large number, and,
but for the sake of this analysis we're not actually going to change anything
on site at the database. We're not going to change any storage clauses. We're not going to
change any table definitions.
We're not going to change any index definitions. And we are just going to use some real-world
principles
to actually identify what the problems are
and then work them out of the system, up the stack
in the application layer, because we believe that's actually where you can make
the most difference.
So, the first thing we want to do because it's all so confusing with all the
clustering is,
we're actually going to start running the system
literally on a single instance.
And just, let's just see what happens there, and
to the system. We certainly expect all those grey wait events
to disappear because we're redirecting all the work load now
down to a single instance. And you can see just by redirecting the workload
to one instance, we actually increase the performance quite dramatically and the
response time has come down.
But you can see that this really isn't
the total answer because we're still not getting that good utilization within a
single instance.
We still got a lot of contention within the system. We got latch contention here
but we also got index contention and buffer busy waits
still within a single instance.
So perhaps we still--although we've moved the workload around
and shuffled it about--we still don't understand the problem.
But let's have a look and think what we're doing with the workload.
We're still doing a lot of load balancing and shuffling
all the data across all the application servers. Now this file
contains all instrument readings for a single serial number
of a process of an object being manufactured.
Likewise it could be your tax return, it could be your phone number,
it could be basically your browser IP address if you need an online system.
So we're basically redirecting all your requests
pertaining to a single element of data or something,
a cluster of data, across multiple application servers.
Now let's think about that if we could actually just get some affinity between
that part
and an application server. Let's think of the impact that that might have on performance.
Getting some affinity up the stack. And instead of trying to load balance
all the manufacturing data for one part,
across app servers, how about just stream it
to one app server? Likewise all requests coming from the browser... instead of
trying to load balance them across multiple app servers
let's try to affinitize it to one app server.
So let's try that. And let's just show affinity
to within an application server.
What we should see is we should see that cross red line
disappear. And now we're starting to see
we are sort of really actually starting to make a big difference to the performance
because we've, we've actually, what we've done is, we've actually
in this case halved the number of processes all fighting
for the same data that was being fused
either between two instances when we were in the rack case,
or contention for the same blocks within a single instance.
So we've taken the performance up from less than
8,000 transactions a second but we're now getting it up into the thirties
and all we haven't changed a single thing in the database at this point in
time.
All we've done is redirect everything to a single instance at this point
and then got some affinity between the application
an application server. Now each application server
has multiple threads and we're still load balancing
amongst those threads within an application server.
So the next logical step would be to get some affinity
between that data and just one application server thread.
So if we were manufacturing a piece of equipment
every bit of data associated with that piece of equipment,
if that was all to go through one application server thread
there would be no contention on that data from any other threads.
So let's try that, and this, and what we will do
what we call full affinity.
And this would be the same as if we had a browser.
So we would redirect and get some affinity that
all requests from a single browser. We'll just go down one application server thread.
Now you can actually see that we're really starting to move and in fact the
red and blue
lines are now showing that we're doing a lot more throughput
on these sort of things. And you can see literally by getting affinity we've
removed also the contention.
And we can see that we've taken the throughput here
literally up to 65,000 transactions. We're nearly ten times faster
than we were when we started, okay.
So this gets to the point...what we've learned that if we stop
stop trying to hash everything over everything and load balance everything
over everything,
by doing all that load balancing what we are actually doing
is we are initiating contention at both app server and database level.
And we are also diluting our ability to get
good cache hits to find the data in your local cache.
So the last move on this move is instead of having
a connection where all app servers connect
to all, all database instances, if we were to map the app server to a specific
instance,
we can cater the next point, and then we can fully utilize all the hardware.
So now we have a dedicated service where each application server
maps to its own database instance. You can see that the performance at this
point,
we've now taken it up to twice what we were achieving with a single instance.
We're now at about 125, 130
transactions per second, which is more than 10 times that we achieved
when we were load balancing everything over everything, okay.
Now it may look smart in an architectural diagram to load balance
everything over everything,
but all data by its nature tends to have some sort of clustering ID,
So if we can root things through the middle tier clustering on that data
element,
now that could be your social security number, your telephone number
it could be a serial number of a part...all those sort of things
or even your browser IP address,
we can actually affect some affinity
at the database level. And it comes back to that old-fashioned idea
if you can be modifying your data in your own local cache
it's a thousand times faster than if you have to get it either from another
instance
or from disk. And you can see
as a result of getting better affinity through the system, we have made the
performance go
from 8,000 transactions a second to nearly a hundred forty thousand
transactions per second.
You'll notice we didn't change anything
on the database. We didn't change the storage clause.
We didn't rebuild an index. We did absolutely nothing at the database level.
We just basically refine the workload
coming through the middle tiers such that we could get good clustering of
the data. Now this doesn't have to be an exact solution.
This is by no certain an exact solution.
But if you can get clustering through the middle tier
and you get it right nine times out of ten, you going to be doing better than you
do it five times out of ten.
If you can get it correct 99 times out of a hundred,
you're going to do better. But you'll no way in modern applications are far too complicated to
ever get this to be exactly 100 percent.
But if we can get it into the nineties, we're doing really well.
And this is really important it is understood that you can kill systems
performance by trending to
hash everything over everything, load balance over everything over everything.
We actually know this in real life. If we keep all the data
that we want to process together together instead of spreading it, out
we always can do that performance. It's very simple.
When we... if we were basically to put all the data we needed over
sixteen disks, it would take a lot longer to process. If you put it all on one disk
you can process it together. We all know this stuff.
So with that in mind, um, you can see we've built a high-performance system
that's doing high
throughput, but it's sensitive. People need to be educated as to what you doing.
Architects need to think about data affinity through the middle tier.
And this can be degraded by literally somebody
not quite understanding what their doing, what's going on.
And if suddenly they decide to say, "oh I'm going to use a different connects string
and perhaps we'll use a
something like the scan list" and they're thinking they can load balance it,
they can devalue the whole thing straight away. Because you can see fairly
quickly,
we're now trying to load balance. We think we're logically load balancing over
instances,
but we're now degrading our cache affinity.
And so back come the grey wait events and the system throughput has been halved.
So you can see by not touching anything at the database,
and just trying to ensure better data affinity coming through
on the transactions, we can have dramatic impacts
on response time and on throughput.