In the last couple of videos, we saw the motivation or the need for the floating point format,
and over here, we’ll take a look at the technical specification for actually representing
the floating point format in a digital system. So as we learnt, floating point representation
is essentially scientific notation, in base 2. So we’d have 2 parts, namely the mantissa
which represents the significant digits of the number, and thereby its accuracy, and
the exponent, which is a number raised as the power of the base of the number system.
The base is 2 here, for the binary number system. And we hinted at a representation
sort of like this, where some bits are allocated to the mantissa and some for the exponent,
and possibly a bit for the sign as well. Now, these could be represented in several different
ways, so for the sake of consistency and portability, a standard exists called the IEEE Standard
for Floating-Point Arithmetic (IEEE 754) which was introduced in 1985. This standard is essentially
a technical specification which governs and defines a lot of things, including arithmetic
formats for different word lengths, interchange formats, rounding rules, arithmetic operations,
and exception handling, all regarding floating point numbers.Today we’re just going to
discuss the arithmetic formats, and representation of floating point numbers, and more specifically,
2 variants. A 32 bit variant called the single precision floating point format, and its bigger
brother, the 64 bit variant, commonly called the double precision floating point format.
These are typically supported in most hardware, and hence supported in most high level programming
languages.
Let’s look at the 32 bit single precision floating point format first. So we know that
32 bits should contain the mantissa, the exponent and the sign bit. So this is how the standard
defines the ordering. The leftmost bit contains the sign bit, which represents the sign of
the number, either positive or negative. The following 8 bits represent the exponent and
the remaining 23 bits represent the mantissa. Let’s start from the simplest section, the
sign bit. Here, we use the sign and magnitude representation, 0 signifies the number as
positive and 1 as negative. This might come as a surprise to you, as we deliberately avoided
the sign and magnitude representation format for integers. Basic arithmetic operations
became more complex and there was the ambiguous zero problem to deal with as well. So we chose
2’s complement instead. But over here, the floating point format is an altogether separate
beast. Floating point operations are fairly complex, and to handle them efficiently, we
have specialized and dedicated hardware called Floating Point Units, which are integrated
into the CPUs of most general purpose computers. Since they do a lot of complex operations,
handling ambiguous zeros is relatively speaking a minor task.
On to the exponent. Now the exponent would have to store negative and positive values
on it’s own, irrespective of what the sign bit holds.But it does it in a rather unusual
way. 8 bits for the exponent could potentially store 256 distinct values, but of these, 2
combinations are reserved for special cases, all zeros and all ones. We’ll see what those
special cases are later. So values from 1 to 254 are available. The arrangement is something
called offset binary notation, where the lowest value represents the least negative number,
the highest value represents the most positive number, and the value in the middle represents
zero. So the actual exponent value is not the value itself, but the value that’s stored
minus a 127. So if you were to store 1 in the exponent field, the interpreted value
is 1 - 127, which is 126. If you store 254, which is the highest you can go to, the interpreted
exponent value is 127. If you actually want to represent the exponent as zero, you need
to store a 127 in this field. I know it’s a bit confusing, but that’s the way it goes.
Let’s look at the mantissa now, we have 23 bits of accuracy for the mantissa. Remember
how I stated that the number in scientific notation should always be stored in the normalized
form. A normalized form is essentially where the radix point, which can either be the decimal
point or the binary point, should always be represented after 1 significant digit, and
this significant digit should not be zero. If it is, like in this case, the exponent
can be adjusted to bring the significant digit to the right of the radix point. In decimal,
the normalized form is achieved if the significant digit is between 1 to 9.
What’s the consequence of normalizing a number for binary. Since binary numbers can
only be represented as zero or one, the significant digit should always be a 1, if we want the
normalized form. For example, this number isn’t in its normalized form, and to correct
that, we adjust the exponent such that the normalized form is achieved. But since we’ve
agreed to store only normalized numbers (and we’ll break this rule soon enough) and since
1 is the only binary value as a significant digit that’ll normalize a number, we can
assume the significant bit to be always one, and we don’t have to store it.
So for instance if we want to store this binary number, we only store the fractional part
after 1, into the mantissa field, so we get an additional implied bit of accuracy, which
gives the 32-floating point number, 24-bits of accuracy, instead of 23.
Going the other way now, if you want to read the number stored in the mantissa, we read
it as 1 point, the value of the mantissa.
So let’s look at a couple of examples shall we? This is the bit pattern stored in a 32
bit floating point number. Can you calculate the decimal value of this? Pause the video
if you can.
Let’s break it down into parts. Now, the sign bit is 0, so it’s a positive number.
The exponent is 128, so to get the actual value, we subtract a 127 from this, which
will result in the actual exponent being 1. Now, let’s look at the mantissa. The mantissa
stores the fractional fixed point values, so each bit represents the value of 1 over
2 to the power of that index. So the first bit represents 0.5, the second is 0.25, third
is 0.125, fourth is 0.0625 and so on. Here, two bits are activated, the first and the
second, so it’s a simple addition of the 2. 0.5 and 0.25, which is 0.75. The overall
mantissa should be represented as 1 point the value stored here, so it’s 1.75. Final
answer is 1.75 times 2 power 1, which is essentially 3.5 in decimal.
If you’re following all this, then great stuff! Let me throw a small wrench into the
equation and complicate it up a little bit. How about this number? Same exercise as before,
pause the video and try it yourself if you can.
Let’s break it down again, The sign bit is 1 this time, so it’s a negative number.
The exponent is 0 followed by all 1’s which is 127 in binary. So subtracting 127 from
it, we get the actual exponent value of 0. The mantissa looks a bit menacing. If you
look at the bit pattern, it’s a recurring series of digits, this part keeps repeating,
as long as there is enough accuracy to represent it. Now, you can go ahead and calculate the
mantissa by calculating the weights of each digit and adding them up, in fact a computer
does this. But you can faintly recognize from the approximations video that ⅕ results
in this repeating bit pattern, so the value that this mantissa was trying to convey was
1.2, but unfortunately, this fraction cannot be represented accurately in the binary number
system, and you end up with a calculated value of 1.19999992847, which I’ll approximate
to 1.2. So the final value is just -1.2, which is not a menacing number at all.
So yes, it’s a bit difficult to get an intuition for floating point numbers in it’s binary
form. But it’s not rocket science, you just need to apply some rules. This website here,
the link in the description is a fun playground of sorts for converting between decimal and
binary floating point formats. You can flip each one of these bits around and see how
it impacts the final number.
Finally let’s talk about range. So we’ll just look at the smallest magnitude and the
largest magnitude possible with the 32-bit floating point format. The smallest magnitude
possible is the exponent having the least significant bit activated. which is essentially
2 power -126. And the mantissa being zeroed out. Technically speaking the lowest magnitude
is 0 itself, but as I mentioned earlier, the exponent having all zeros or all ones is a
special condition which we’ll talk about later. So the lowest magnitude, with what
we know now, is 1 time 2 power -126, which in decimal is around 1.18 times 10 power -38.
Now the largest magnitude this format can represent is all but 1 of the bits activated
on the exponent, and all of the mantissa bits turned on. You get very close to 2 times 2
power of 127, which in decimal is around 3.4 times 10 power 38.
Now you can’t represent fractional numbers at this end of the range, and it makes sense,
it’s not a magical format to give you fractional accuracy for such large numbers, and it doesn’t
really matter to have fractional accuracy for such large numbers anyway. But for smaller
numbers, you do need it. The accuracy scales along with the range. At any given point in
the range, the difference between any 2 consecutive numbers is around 1 / ten millionth of the
value of the number. That seems like a shot in the dark, but I’ll dedicate a video on
accuracy a little later.
Now, whatever we discussed for 32 -bit single precision floating point numbers, you can
apply the same to 64 bit double precision floating point numbers. The only difference
is that you have 64 bits to play around with. The standard defines that the double precision
format would have 1 bit for the sign as usual, 11 bits for the exponent which gives you a
monumental range, and a massive 52 bits for the mantissa, which gives you much more accuracy.
So the double precision floating point numbers give you extra accuracy and extra range. The
extra range is not really useful for most domains of work, I mean look at it, it’s
massive. Both the examples I give in an earlier video, the mass of the earth and the planck’s
constant can both be represented fairly easily in a single precision format, you really need
to look hard to find numbers beyond this range. But what double precision numbers offer more
than anything else is accuracy. The 52 bits of mantissa gives it a lot of resilience to
errors that may accumulate in arithmetic operations.We’ll talk more about accuracy soon enough, but
we’ll finish off the special cases that surround the floating point format.