This video is all about trying to visualize floating point numbers.
We checked out the floating point format in the previous video.
And it was really just the technical specification of the format that we discussed.
Floating point format is an objectively difficult system to understand and visualize.
Primarily because this is how we represent the floating point format, using the sign, mantissa
and exponent notation. And we derive the resultant floating point number using this formula.
Sure, this looks complicated, it’s not very intuitive.
But the difficult part to visualize is what this formula isn’t telling you. What this formula isn’t
defining is the precision of the floating point numbers and the least count between them.
Let's do a quick comparison between the 32-fixed point format, and the 32-bit floating point
format. Let’s look at range specifically. 32 bits can represent 2^32 distinct possible
states. An unsigned 32 bit fixed point number can store numbers from 0 to 2^32 - 1.
This is intuitive and easy to grasp. The difference between each number is 1. But what
about a 32 bit floating point format? We can store numbers from around 2 X 2^127 to - 2 X 2^127.
That’s a monumental range, way bigger than what 32 bits can store. How is that possible?
Obviously this isn’t some magical format. There is absolutely no way to have this massive range,
within this limited 32 bit space, if we were to represent each one of these numbers distinctly.
There has to be some sacrifices made so that we could represent the whole range
with an acceptable loss in precision. Now that’s the interesting question. Where is this
loss of precision? Where does this kick in? What does it look like?
Ok, let’s step back a little. We keep talking about precision here.
What exactly is precision. Accuracy is another term quite often used in tandem with precision.
Are they the same. Not really, no. Accuracy is how close the result is to the real value,
the mathematical truth. Precision is how specific the result is.
Still quite unintuitive, but let’s take a concrete example. Let’s say you want to
measure the length of a car. The mathematical truth is that the length of the car is 255cm.
But you’re given a ruler that only measures in kms. You could say that the car is 132.327521km
in length. Now this is wildly inaccurate, since it is nowhere close to the mathematical truth.
But it is quite a precise measurement, since it is very specific, and has a very fine least count.
Now I could also make the measurement and conclude that the length of the car is
0km in length. Now this is an imprecise measurement,
since it’s not very specific. But it is a lot more accurate than the previous measurement.
The precise and accurate answer to this question would be 0.00255km.
Now let’s take a look at the number 2, in fixed point format. What’s the next
number that’s representable in this format? It’s 3 ofcourse. The difference between each
consecutive number throughout this entire range is always 1. What about the float point format.
What’s the next representable number after 2? It’s 2.00000023842. Now, we’ll get to where this number
comes from, but you can see that loss of precision is not a problem here for floating point format.
It’s got a really fine least count, miles better than that of the fixed point format.
The next number after that, is just an addition of this precision value. And the next one after
that is the same. So far, the floating point format is outshining it’s rival.
Let’s look at a bigger number. This one 1073741824. This is 2^30 by the way,
and this can be represented in the 32 bit fixed point format. What’s the next representable value
in fixed point? Well you’ve heard it already, the difference between consecutive numbers is always 1
in fixed point, so the next number is 1073741825. What about in floating point though? The next
possible representable value is 1073741952. It’s 2^30 + 128. The least count is 128. numbers in
between are unrepresentable! That’s a massive loss in precision, wouldn’t you agree?
Let’s try and understand how precision varies in the floating point format.
The following section is inspired by a blog entry written by Fabian Sanglard, called
Floating Point Visually Explained. The link’s in the description, and it’s an interesting read.
Instead of looking at the floating point format in terms of exponent and mantissa,
we can instead look at it functionally, as to what the exponent and mantissa actually
represent. We can call them Window and Offset. The Window represents a window of numbers or
a range of numbers. This is always a consecutive power of 2 numbers. Like 2 - 4, or 4 - 8 or 8 - 16
or 1024 - 2048 or 2^30 - 2^31. It represents a range of numbers essentially.
The offset pinpoints the exact position within the window of numbers.
Since the offset section is 23 bits in length, the window can be split up into 2^23 distinct numbers.
This is constant for the 32 bit floating point format, it doesn’t change. All we are saying is,
we are splitting the window into 2^23 intervals. The smaller the window,
the smaller the interval, and finer the precision. As the window size becomes larger,
each one of the 2^23 intervals gets larger, the precision is more coarse and granular.
Let’s calculate the precision. All we do is divide the total numbers in the range by 2^23. When the
window is 2 - 4, the precision is this tiny number 0.000000238. When the window is 1024 - 2048,
the precision is 0.00012. It’s growing and getting worse. When the window is between 2^30 - 2^31,
the precision is what we encountered earlier, it’s 128. All the numbers within a window
have the same precision. Once we have reached the maximum representable number in a window,
we can "float" the number to the right and represent the number within the next window.
The cost of doing that is the added loss of precision.
Fun fact, that all numbers within the window of 2^23 to 2^24 have the same precision as that of
fixed point, the precision is 1. Below this range, the floating point precision is better
than that of fixed point. And above this range, the floating point precision is worse off.
Finally, I’ll talk about one last way of visualizing floating point numbers.
It’s through a ruler. Let’s say we have a 1m ruler. The least count on this ruler is 1mm,
so it has 1000 markings on it, so we have 1000mm, which is just 1m. Great. If you want to measure 2
distances spaced apart, then you can. If you want to measure the distance of the objects
spaced close together, you can measure this too, with the same level of precision every time.
If you want to measure two large distances spaced more than a 1m away,
then you can’t measure it. You need a bigger ruler.
This is analogous to fixed point representation. The ruler is fixed, so is the precision. The
representable range is limited to what the container or the ruler in this case offers.
What about floating point? In this case, you can think about the ruler itself scaling to meet
the needs. If the distances to measure are close together, the ruler could shrink down. You still
have 1000 demarcations on the ruler, but they are closer together now, and so the precision
with which you can measure the distance is higher. If the distances are far apart, then
the ruler could grow in size. Again, there would still only be 1000 intervals in the ruler, but the
distance between the marks increases. You could measure the distance no matter how big it is,
but you can only measure them with a low degree of precision. This may seem like a great tool,
the scalable ruler. And it is. It absolutely is! But there are limitations. For instance, you can’t
use the same ruler to compare two distances. A really small distance and a really large distance.
The small distance would be too small to be even captured by the limited precision that
the ruler is operating at when looking at large distances. We’ll explore these limitations with
real world examples soon enough. But before that, let’s just summarize what
we know. Fixed point numbers give us fixed, predictable precision. With floating point
numbers you can store either very large numbers imprecisely, or very small numbers precisely. If
you remember just this fact, then a lot of things become apparent. I’ll see you in the next one.