Hey I'm Dave, Welcome to my Shop! I'm Dave Plummer,
a retired operating system engineer at Microsoft going back to the MS-DOS and Windows 95 days,
and I wrote the Windows Disk Formatter that we've all been using for the last 25 years or so.
And today, I'll reveal the secret history of Windows Format.
This is the actual memory card that I used to write and test Windows Format
back in the day on the Windows Shell team. What's on it? I don't know, I haven't looked, I'm a busy
man. But it was huge for its day, coming in at 16 megabytes of rated capacity. That's enough
to hold two copies of the e-book for Steven King's the Stand. No joke, I looked it up. It's 7 meg.
This is a 256 gigabyte micro sd card. Super handy for dash cams, drones, security cameras, and more,
but if you want to format the whole thing under Windows, you can't. Do you know why you can't?
Because I said so. Yeah, that's how my kids look when I say stuff like that. But
it's literally true in this case, and that's today's story. I was working on porting the
Windows 95 shell over to Windows NT and making it run on MIPS, Alpha, Power PC, and so on.
I guess perhaps because I had worked on smartdrive and other block-level device stuff before,
I was tasked with writing the new Windows format. In this case it had to be a replacement
and complete rewrite since the Win95 system was so markedly different - we had multiple
filesystems and compression and encryption, and we allowed you to vary the geometry and so on.
There was no equivalent in Win95. Now I didn't write the device code,
the parts down in the driver and filesystem that actually do the zero'ing of the bits on the media,
but rather, the part that the user interacts with to set up how the format will be organized, what
filesystem to use, how big, the geometry in terms of clusters and sectors and tracks and so on.
And up at the very top of this software stack sits the actual Windows Dialog that the user sees.
It's only a fraction of what I call the UI, and I wrote that part as well as everything else all the
way down to the API level. I think I wrote that Dialog on a rainy Tuesday morning between 9AM and
10AM. Which is my way of saying that I did not spend a lot of time on the actual layout of the
user interface, because I'm not a user interface designer by trade. We had very talented people,
including Joe Belfiore's entire Win95 design team, that did such work, and other than Task
Manager and a few other things, I largely stayed in my own lane when it came to design work.
Plus, if I had my way everything would look like Visual Studio anyway, or so I've been told by
those who should know. This is the interior for my Range Rover. I've kept the old boy at
least ten years in part because I like the user interface more than the newest ones - this old one
is entirely hard buttons. The LCD display provides info and feedback and runs the infotainment, but
everything about the physical vehicle itself is located on a hard button or switch or knob.
That's the way I likes it. If you've ever fumbled for the hazard switch in a rental car then you
can imagine how much worse it would be if you had to navigate through the vehicle and safety menus
first to get there. That's why the hazard button is a hard button in every car by federal law.
That's part of the reason why my wife's Tesla can be a little frustrating to me; with hard
buttons you know what selections are available to you, whereas with a dynamic user interface
like the Tesla, clicking on a button might not do anything more than show five more buttons.
Which is why Format is basically a vertical stack of buttons and dropdowns that shows all your
decision points onscreen at one time in a single window. It's not fancy, and it's not elegant,
but it's all there for the world to see, including the hypothetical designer who should be along any
moment now to design a beautiful and functional user interface for Windows Format. At that point,
on sheer nostalgia alone, you'd imagine Microsoft would want to get the band back together so badly
that they would just back a truck of cash up to my house in order to get me to come out of retirement
and actually code their new dream user interface for Format. But, since I've been sitting on the
front steps waiting for that to happen for about 20 years to no avail, let's just assume that it
won't be happening, and that Format will stay largely as it is for the foreseeable future.
It's not pretty, but it gets the job done. Mostly. Except when it comes to FAT32. To explain why that
is, it helps if you know a bit about what it means to format a disk and why we do it. It's not simply
erasing the disk by any stretch. As you might know, there are really two steps required to
prepare a disk or a memory card for use: first you partition it, and then you format the partition.
Partitioning is the process of breaking a big disk up into smaller volumes. For our sake though we'll
keep the conversation simpler and just assume every disk is partitioned into one big volume.
Then we just have to format that one volume. That's what most people do with removeable media
anyway. Formatting is the process of laying out the structure and geometry of the disk.
It logically - and in some ancient cases magnetically - carves the disk up into
tracks and cylinder and sectors and so on. In the olden days you even had to know how many tracks,
like and old LP record, were on the disk, and how they were in turn split up into sectors.
Today you just find out how many sectors are in a partition and that's largely it.
The drive hides or abstracts away the physical reality and you just deal with a bunch of
sectors. That's doubly handy because the drive can also keep some sectors spare in case any go
bad and remap them without you ever seeing it. The FAT32 filesystem keeps track of the disk in
clusters, and it keeps track of which file is using which cluster, and so on. To number all
of these, it organizes the bytes on the disk into 512 byte sectors which are then grouped
into clusters. The trick is that no matter how big the disk is, you can never have more than 4177918
clusters on a disk. We won't get into the technical reasons of why, but that limit is
fixed forever for FAT32. So, if you divide a disk up into 4 million clusters, the bigger the disk,
the bigger each cluster. And hence clusters are actually groups of bytes, up to 32K in
size. And that's the problem. It's called Cluster Slack. It's wasted space, and FAT32 is bad for it.
Why? Because regardless of the file size of your file, it has to go on the disk as a list
of clusters, so the size is always rounded up to the next full cluster size. That means if you have
a tiny file that just says "Hello World", and you put it on a FAT32 disk with 32K clusters, instead
of 12 bytes it will take up a whole cluster of 32768 bytes, which is a waste ratio of more than
99.9% of your disk! And this happens for every file no matter its size, so the amount wasted
can add up very quickly with large clusters, particularly with lots of small files. We call
it cluster slack, and it is the unavoidable waste of using FAT32 on large volumes.
How large is too large? At what point do you say, "No, it's too inefficient, it would be folly
to let you do that?" That is the decision I was faced with back on that rainy Tuesday morning.
There's a famous apocryphal story wherein Bill Gates is quoted as saying that 640K is as much
memory as anyone would ever need. Unfortunately, cute though it is, that story isn't actually true,
as he never said it. And besides, 640K wasn't a limit that Bill picked,
it was a technical reality imposed on them by the Intel architecture. But the FAT32
decision wasn't like that. You could, in theory, create a FAT32 disk of enormous proportions,
terrabytes in fact. But the cluster slack would be so enormous that it would waste gobs of space.
I imagine my thought process went something like this: The memory card I was using for testing,
you might recall, was 16M. That was the largest I could get my hands on. Perhaps I multiplied its
size by a thousand, and then doubled it again for good measure, and figured that would more than
suffice for the lifetime of NT 4.0. I picked the number 32G as the limit and went on with my day.
I didn't start to regret that choice until SD cards got to the magic 32G size many years later,
and then it started to bite me as well. Until now I haven't explicitly stated that I thought I was
only picking the limits for NT4.0, and only until the designers revised the UI. In other words,
I thought my work and my decisions were somewhat temporary. That, however, is a fatal mistake on my
part that no one should be excused for making. With the perfect being the enemy of the good,
"good enough" has persisted for 25 years and no one seems to have made any substantial changes
to Format since then. Some have speculated that restricting the FAT32 filesystem was
a sinister plot to promote the adoption of NTFS. That's silly for a couple of reasons,
not the least of which being that every system disk has to be NTFS now anyway,
so there is no shortage of NTFS penetration. It's on every system already. Furthermore,
NTFS isn't always appropriate for removeable media, so Microsoft had to include FAT32 support
for some types of disks whether they liked it or not. As far as I've ever seen, they've never
promoted or licensed NTFS to consumer electronics makers. A more believable scenario, except one
that I can also assure you did not factor into my decisions, was the future release of exFAT which
can accommodate enormous disks. In fact, exFAT can support disks up to 64 zetabytes in size,
with 32 megabyte clusters. Now imagine that - the cluster slack on a simple readme.txt file
would be more than twice the size of the disk that I used to write format in the first place. So you
can imagine numbers of that scale would have been a little hard to forsee as reasonable back then.
You still face the issue of egregious cluster slack, but it really depends on what you're using
the SD card for. I supposed I imagined you might be storing thousands of songs, and wasting many
megabytes thousands of times is clearly wrong. But if you're recording long segments of 4K video,
you might only have a small handful of files on the card, and so the cluster slack is really not
an issue. In the days of 16M cards I didn't anticipate a need for files of that size.
NTFS supports a wide range of features that FAT does not, but if you don't need those features,
and are just building a video camera or similar, then exFAT is probably the way to go. The only
downside is that Microsoft opted to license it, and kept the source private - they license the
spec to independent software and hardware vendors for use in their own products. I don't know the
pricing details, but I suspect it's not cheap, since it's aimed at companies like Sony and JVC,
not mom and pop shareware vendors. I've heard $300K but that could be old news.
So where does that all leave you if you have a large memory card and your device doesn't
support exFAT? Are you really limited to 32G in size? Well, using the built-in dialog, yes.
But thankfully there are two easy ways around it. The first is to just use the command line,
which allows you to specify geometry in excess of what the shell dialog allows, so you can make a
disk as big an inefficient as you'd like, subject to the FAT32 limits. The second alternative is to
download a utility like RUFUS or FAT32FORMAT that bypass the shell dialog and calls the
filesystem directly, just like the command line tool, so they are not hampered by such limits.
At the end of the day, it was a simple lack of foresight combined with the age-old problem
of the temporary solution becoming de facto permanent. I don't honestly know why they haven't,
over the years, expanded the limits right in the shell dialog - I know I've suggested as much to
the right folks. It could be a way of ensuring the fat cash from the exFAT licensing gravy train
keeps on flowing, but I highly doubt it. If it were me, and it used to be me, I'd be pretty
scared of suddenly allowing people to create disks of a size that have never been tested in more than
two decades of consumer electronics! Somewhere out there is a camcorder that bricks or erases your
wedding video when you put in a 64G FAT disk, and no one wants those phone calls, especially when
then win is so small and a well-tested solution in the form of exFAT already exists? Do I wish
it were free? Sure, but I wish a lot of things were free that aren't. But in this case wishes
do sometime come true, because in the last year Microsoft has opened up the spec for exFAT
and officially supports the addition of an exFAT driver to the Linux kernel. How that affects the
licensed version I do not know, but an unofficial open-source version can't be far away now.
So, there you have it: the problem, the why, the workaround, the solution, and the future
all wrapped up in minutes. If you like these little anecdotes about ancient Windows history and
why things are the way they are, be sure to check out my Windows War Stories playlist on the channel
and do consider subscribing to Dave's Garage so that you get notified of future episodes.
I'm not selling anything, and I don't have any Patreons, I'm really just in this for the subs and
likes, so please leave me one of each before you go! Thanks for joining me here today in Dave's
Garage. In the meantime, and in-between time, I hope to see you next time.