Let's look at a single cache that is 16 kilobytes in size.
Let's look also at a single cache that is 128 kilobytes in size.
Let's look at no cache, simply we will just access the main memory, and
let's look at the hierarchy that has this kind of cache as the first level
cache, and this kind of cache as the second level cache.
We will look at the the hit time for this, the hit rate for
all of this, and the overall AMAT.
So a 16 kilobyte cache might have a hit time of 2 cycles,
a hit rate of 90%, so it gives us an overall AMAT
of 2 plus 10% of the time we have the memory latency.
Let's say the memory latency is 100 cycles.
In that case, we end up with an overall AMAT of 12,
2 from the hit time and 10 on average from the miss penalties.
Now let's look at the larger cache.
This cache might have a hit time of 10 cycles, but
a hit rate of maybe 97.5%.
This is a bit high for such a cache, but let's say it is.
The AMAT when this is used alone would be 10 plus
the miss rate times the miss penalty, and
we end up with 12.5, which is a little bit worse than with a smaller cache.
So having a larger cache alone increases the hit time, improves the hit rate.
So it may or may not improve the AMAT, but
either way the AMAT is not going to improve a lot.
Of course, having a cache is still lots better than not having a cache.
Not having a cache means that our memory has a hit time of 100 cycles,
basically the memory latency.
It hits 100% of the time.
So AMAT ends up being 100 plus 0 times the penalty, because it never misses.
But we still end up with 100 cycles per axis, which is way too big.
So obviously, having a cache is better than not having it.
So now let's look at the cache hierarchy.
The hit time is different depending on which of the levels we hit in.
The hit time is still going to be two cycles for a level 1 hit.
For a level 2 hit, however, we first check in level 1.
So we pay 2 cycles to check.
Then we access L2 and have a hit.
So overall now, it's going to be 12 cycles for a level 2 hit.
The hit rate is going to be 90% for L1.
It's still the same cache as if it was alone, and
of all the axises that go from level 1 to level 2, we will have
some hits because the L2 cache has this hit rate when working alone.
Of all the processor axises, it would have a hit on this many, but
of all the processor axises,
those that hit in L1 probably would also have been hits in L2.
So this cache really has a 75% hit rate of all the things that go into it.
It would have this hit rate if it was alone, but the L1 cache is filtering
all of the easy to hit axises, so the L2 only gets 75%.
Basically it only gets to see kind of the worst 10% of the axises, and
for those it's only having a three quarters hit rate.
But the AMAT is now going to be two for L1 hits
plus 10% of the time we have an L1 miss.
When we do, we have 10 cycles to axis L2,
plus 25% of those also end up accessing memory.
So when we compute what this is, it's 10 plus 25.
So the overall miss penalty for our level 1 miss is just 35 cycles.
It's much better than what it used to be when the L1 was working alone.
And we finally end up with 2 plus 3.5,
which gives us an overall AMAT of 5.5 cycles.
This is much better than either of the two caches working alone.
And it's much better because really we are having most
of the axises hit using the hit latency of the fast cache.
Some axises have a higher hit latency in the second cache,
and fewer of them made this huge memory latency.
So this is why cache hierarchies work better than individual caches,
and why we have them.
It is not enough to simply have a large cache, if it's also slow.
In that case, you can combine their good properties so
that most of the time we get this hit time.
But overall, as far as the memory axises is concerned,
we really have this hit rate in the combined cache.