Let’s imagine and visualize the precision of the 32 bit floating point format.
If you have trouble visualizing what I’m about to say, I’d encourage you to check out the video on
Visualizing the Floating Point format. Link in the description. So, in it, I concluded that
large numbers are stored less precisely. They are thinly spread out and are far and few in between.
As the magnitude of the numbers decrease, the precision gets better and the least count gets
finer. As you reach the bottom of the range, the difference between consecutive numbers gets
smaller and smaller,as they get ever so close together. Most of the representable floating
point numbers appear close to the origin on a number line, and only a small portion of the
representable numbers appear outwards. Obviously, the same is true on the negative axis as well.
Infact, there are the same number of floating point numbers from
0 to 2 as there are from 2 to the maximum floating point number number of 3.4 * 10^38
This boggles the mind, visualizing this is so difficult. This illustration is not scale,
and it just can’t be. That’s because we’ve been looking
at it on a linear scale. Let’s see what happens when we convert it to an exponential scale.
Here the numbers are represented as powers of 10. from a minimum of 10^-38 to 10^+38.
And we can see that the number 2 is somewhere here, between 10^0 and 10^1.
And now, the original statement, that there are the same number of floating point numbers
from 0 to 2, as there are from 2 to the maximum floating point number of 3.4 * 10^ 38, makes a
lot more sense, and gives a lot more clarity. So, clearly the floating point numbers are
logarithmically arranged. But there is one striking problem with the statement I said.
The first range I mentioned was between 0 and 2. Where is the number 0 on this scale?
You can’t find it, it’s an exponential scale, you can keep going lower down to zero,
but you’ll never quite reach there. A non zero number raised to the power of anything, can never
be zero. The best you can say is that it tends towards zero. Well, at this point of time, you
might be asking what’s the point of all of this, what does it have to do with subnormal numbers.
There exists a chasm between the lowest representable floating point number,
and the number zero. A chasm that our number format can never represent values in.
The number zero itself is a special condition within the floating point format. It’s not part
of the floating point range, so none of the floating point rules apply to the number zero,
it’s a specially handled case with its own set of rules.
Also, the exact same thing happens on the negative axis as well, when the sign bit represents
negative numbers. There’s a chasm or a void between the lowest negative number and zero.
Also, a look at this scale, it’s discontinuous, there’s no continuity between the positive range
of the scale and the negative range of the scale. The format of numbers is discontinuous
with a little tiny island of the number zero that we’ve introduced for the sake of mathematics. And
this is where subnormal numbers come in, their purpose is to bridge this gap, to fill the void
so to speak. Ok, interesting,
but why do we need to fill this gap? What’s the motivation for doing this?
We’ll take a look at the 2 numbers. The minimum representable floating point number, which is this
number. And the next number after that. Since these are really small numbers, and based on
the fact that the smaller the numbers are, the more precisely they are stored, you can imagine
how small the least count is going to be. Well, we don’t really have to imagine, we can calculate it.
Subtracting these two numbers will give you the least count. This is clearly lower than the lowest
representable number as a float. Far lower. What would happen in this case? How would we store the
result of this subtraction operation? We’ve just hit an underflow condition. Any value lower than
the lowest floating point number is flushed to zero. Though the answer is technically not zero,
you could say that’s alright, because these numbers are so small. But too small for who?
We might think it’s small, but a mathematician or a scientist may not. Treating arbitrarily
small numbers as zero could ruin the integrity of calculations, and could result in unpredictable
and undefined outputs for different input values. So what the subnormal numbers guarantee
is that addition and subtraction of unequal floating-point numbers never underflows;
two nearby floating-point numbers always have a representable non-zero difference.
Let’s look at how subnormal or denormal numbers are implemented.
We’ve got the floating point format with the sign, exponent and mantissa. How do we
represent the magnitude of the lowest floating point value? The sign is unimportant here.
The exponent has to be one, which represents an exponent value of -126.
The exponent cannot be zero, because it’s reserved for special conditions, including the condition
to represent the number zero itself, where both exponent and mantissa fields are zeroed out.
Ok, back to our lowest representable float, the mantissa can be zero’d out. This means the
number represents 1.0, since the 1 before the binary point is implied, and we get a
normalized representation of 1.0 * 2^-126 in binary, which is 1.175e-38 in decimal.
All good. But, there is still a reservoir of numbers that we haven’t utilized yet.
When the exponent drops down from 1 to zero, we trigger the special conditions,
but there is technically 2^23 bit states of the mantissa within this window that are unused.
One of them, ofcourse is used, and its to represent the number zero. And so, we can
use the rest of the 2^23 - 1 states to bridge the gap between this number and the number zero.
We can’t keep the normalized form here, with 1 point something. If we do that,
we are just adding another level of precision down, and not solving any of the problems. So,
we have to take the 1 off here, and represent the numbers as is, with 0 point something. As
we can see from the illustration, we can gradually underflow the numbers to zero. In this form, the
number in the mantissa is no longer in scientific notation, it's no longer normalized. It’s been
denormalized, and that’s where the name for these numbers come from, denormals or sub normals.
In this special case, the exponent still stays at 2^-126. It doesn’t drop in value.
Right. Let’s see what the lowest subnormal number possible is.
It’s when the exponent is all zeros and the mantissa is 1 on the least significant bit. Yea,
this makes sense right. Let’s find out what the value is in decimal.
We can manually calculate this or let a program calculate it for us. Let’s go back
to the program. In the numeric limits class, we previously got the minimum normalized value. But
we can also get the minimum denormalized value as well. Let’s print that out.
There are 2 really important facts that we can get out of this.
One, the difference between the lowest normalized numbers, or rather the lowest
precision value that we can get is the exact same as the lowest denormalized number.
This is quite profound, once you realize that the subtraction between two non-zero,
non-equal numbers will always be non-zero! Still very cryptic, but in simple terms,
it can be mathematically proven that subtracting any 2 distinct floating point numbers
will never give you zero in calculations, it’s always give you some result. This
result may not always be accurate, but it’ll give you a result nonetheless.
The second point to note, is that the lowest denormalized number is about 7 orders of magnitude
lower than the lowest normalized number. How is this possible. What kind of witchery is this?
How are numbers stored accurately within this huge range, within 1 window of
numbers. The simple answer is, the accuracy of subnormal numbers is abysmal, it’s really bad,
and shouldn’t really be compared with floating point accuracy.
Let me demonstrate this. I’ll take the number 1.4 * 10^-38. and I’ll multiply this number by 1.5.
This is well within the normalized range of numbers, so we should expect to get decent
accuracy here. So, 1.4 times 1.5 is 2.1, so we’re expecting some number close to that.
we can see that the result is 2.1 followed by a few zeros, and some inaccurate numbers there,
so it’s accurate to about 7 decimal places, and this is totally
expected for floating point number. 2.1 cannot be expressed exactly in binary,
and the accuracy is acceptable. What about this number. 1.4 * 10^-45.
Now this is very close to the lowest subnormal number. Let’s multiply this by 1.5.
What do we get? Good golly miss Molly, it’s 2.8. That’s not even close to accurate.
What went wrong? 2.8, what can we infer from this. It’s 2 times
1.4. So it’s twice the number we asked for. This is also the second smallest subnormal number.
So no values between these two numbers can be represented. It is also, the difference between
the third lowest normalized number and the lowest normalized number.
What’s the third largest. Well it’s 3 times 1.4. What does this remind you of?
Clearly the subnormal numbers are arranged in fixed point format. It’s like asking what’s 1.5
times 1 in integer. Well we know that the answer will be 2. Because 2 is the nearest number to 1 in
integer fixed point, and the answer is rounded off. The same thing is happening here.
So we find that the floating point format is a real oddity, it’s full of interesting nuances.
A large portion of the numbers, normalized numbers are logarithmically spaced out,
with floating point representation and precision, and then a small subset of numbers,
denormalized numbers are linearly spaced out, with fixed point representation.
Because of the subnormal oddball, it’s difficult to implement operations on subnormals within
the hardware floating point unit. Some of the processors do it directly in hardware.
But a lot of them delegate the handling of subnormal values to system software. This will
most definitely cause performance penalties for applications that are computing subnormal numbers.
Floating point calculation which could’ve been done within a couple of clock cycles in hardware,
could take as many as 100 additional clock cycles to complete when it’s delegated to
the system software and iit can’t be directly done in hardware.
There is more to talk about regarding subnormal numbers. This is one of the more controversial
additions to the IEEE 754 specification. Not everybody was on the same side of the argument.
Logically and mathematically, it makes sense, and in an idealistic world it makes the format
more robust and resilient. But for reasons I can’t fathom, they are a nightmare for hardware chip
designers and for practically implementing them. We’ll definitely revisit subnormal numbers when we
talk about the problems associated with floating point numbers. But till then, let’s carry on.