[intro music]
[theme music]
Hi. Today we are going to talk about
Linux memory and the problems associated with huge
pages. Huge pages in Linux
are extremely important to use. The more mighty machines actually
looks at them. And systems built without
using huge pages we are more convinced
that you must use huge pages. And we do appreciate that they are
difficult to configure.They take a bit of time, a little bit
planning and strategizing, and figuring how much memory you want to allocate
to huge pages. And then you have to reboot the systems actually to
get the huge memory--page memory. But
in reality, we're just finding that the systems are better behaved
more stable, more predictable,
and this in itself warrants the use of using huge
pages. And a lot of people ask me, "what are Linux huge pages?"
Well, conventional pages in Linux
are actually quite small, and what actually happens is this
shared memory region of your SGA
is becoming ever-increasing
piece of memory. And when a user presses
connects to the SGA, they have to map between their
virtual address space and physical memory which is where the
shared memory is actually located. And this
mapping set of entries, are
often called TLBs, are pieces of memory
that are taken out of the Linux kernel. Now if the pages are
quite small, you can imagine the number these entries
is quite massive, and they occupy quite a bit of memory themselves.
Now the whole concept of huge pages is make the page bigger
so you actually have less TLB entries.
And so you allocate less memory at the TLB entries and so you can actually,
basically map in the SGA without doing
as much work in the Linux kernel, and taking less Linux kernel memory.
And, whilst this seems all very complicated and very
computer sciency-like, it's actually not that difficult
to understand when you sit down with the mechanics
but rather than to give you a lecture on the basics of
Linux memory we just would actually like to show the
impact on systems that have been configured
with huge pages and those that have been configured without huge pages
so you can see pathological race conditions.
And like many Real World Performance
demos that you're able to see, we built this demo, is
because we kept on saying it. We didn't like to get on
conference calls and explain to people why they should do it
and it became a debatable item. As far as real performance is concerned,
Huge pages should be used at all times.
Hopefully these demos will help explain it to you and
understand what goes on when you run out of
memory on Linux. And perhaps, you know, we can
educate you and perhaps we can save
you from taking a system outage. Okay, so on the demo here
you can see we have two systems, one built with huge pages,
and one built with conventional pages. And
what we've done is break up the memory regions in these middle sectors where we see
the pink.
and, and the green stripes, to give you an indication of
where the memories has been allocated. Now I just increase the connections to
nearly 3,400, on both systems
and thing to notice is that whilst
on, on the huge pages there's been an increase
in the pink area--the amount of memory allocated by the process space--
there is still a certain amount of free memory on the system.
Now when I do conventional, I haven't use huge
pages, you know that, you see notice the bump in the pink area
but what we will also see another bump in this green area.
And those are these page table translation pages I spoke
about earlier. And the memory being allocated inside Linux kernel,
associated with those things those pages. Now
the other thing to also learn about is we don't allocate those entries until
the shared memory is being referenced.
So just connecting all people to the database
will not necessary allocate all those entries.
They have to say connect to the database and start doing work.
So as you notice, that as we run this workload
the amount of free memory is diminishing and diminishing
all the time. And this area of thick
ever becoming thicker green-line, so when I started and had all the
connections in, the page tables with 13.7 gigabytes
now at this point in time it's got 27
gigabytes and its ever-increasing. So if we actually look at the
conventional memory model and we see how much memory is free,
you'll notice that the amount memory free is dropping all the time.
We're not seeing that on the huge page case that's just sitting there statically,
okay. And at this point in time you'll notice that we're eating memory
and we're down to about 1.2 gigabytes of free memory.
Now I know in about a second
at this point in time, we're going to trigger a point where we run into memory starvation.
And at this point in time, the Linux page demon
is likely to wake up start having to swap
pieces memory out. And we'll see this manifested
in a format, that basically we start to see the performance degrading.
And we're starting to see this in the pink line with the system degrading to the
point that the performance almost dies. But you notice, we're still
keeping the CPU busy and it's starting to race
as the page demon does some work. And now
we are starting to get into a stable instability where
suddenly it
is reclaiming memory and suddenly,
and it's able to run a bit. And then it runs out of memory.
The page demon goes crazy. The throughput drops.
And it's unable to write. We are in an unstable situation.
You can see this w-shaped curve up here,
where the system is bouncing between stability and instability.
And the more it runs, the more page table entries we take.
And then it renders it unstable and the page demon,
kicks in and tries to claim back more memory.
So you can see that we're bouncing between stability and instability
all the time. And to the point that the throughput sometimes is
zero, sometimes it's not zero. But this is
what happens when we run out of memory on Linux. Sometimes it's more pathological,
and if I was actually increase the number of user processes just a little bit more,
I probably would get to the state
where we're continually swapping memory out.
And we've driven the system into the ground at this point in time.
Interesting to note that the huge page case has hardly changed at all.
This very little change in the huge pages just remain stable,
and now we've driven the system completely into the ground,
and then its re-flamed, it's got back to a 1 gigabyte memory.
And it's coming back to life again. Likewise, I can remove a few processes
and free up a lot of memory, and usually the performance comes back.
The problem is we just cannot predict
when things are going to go stable or unstable when we run out of
memory. And so if on your Linux systems you're getting
periods where you've got a large number connections--a large SGA--
and then the system just periodically appears to just stall,
you're probably running out of memory and it's
starting to swell. And then it may recover itself
and then the system looks great. But you can't predict when it takes place.
The other thing is connecting to the database doesn't necessarily mean will
take the
memory. We have to touch, start touching enough
pages to take it up. So dropping the process count in this place, I've
freed up the system and now we're nearly at 7 gigabytes free memory
by itself, but we are still continuing to eat memory
even though we free up memory. And it will continue to eat it
and as these processes are running. You see, it is starting to diminish
again. And again, I can literally add
a few thousand processes, a few hundred processes again,
which will cause the memory to get allocated again. You can see the
thin green line that we freed up the memory here, will be diminishing
very, very rapidly,
and we can knock the system over again.
Down to 3.4 gigabytes of memory rapidly falling again.
3.2
3
We've killed it again. I'm just saying we can just literally run out of memory,
and the erraticness of the pink line,
and the spike in the CPU, you can see the page demon is
coming in--trying to free up memory. And we are seeing memory getting
freed up, and then the performance comes back because, and then because the
performance comes back, we touch more pages, and then the page demon has
to kick back again.