Hey I'm Dave, Welcome to my Shop!
Today we're putting the Windows shell progress dialog to the test.
Love it or hate it, I used to own the progress dialog when I worked at Microsoft.
Recently I was on Facebook, of all places, when someone sent me a cartoon from one of
my very favorite comic strips, xkcd.
That's nothing new, but the fact that I was apparently the SUBJECT of this cartoon is
what made it interesting to me!
Let's have a look:
The Author of the Windows File Copy Dialog Visits Some Friends
I'm Just outside town, so I should be there in fifteen minutes.
Actually, it's looking more like six days.
No wait, thirty seconds!
I love it.
It's funny because it's true, and because I can take a joke, even if its somewhat at
my expense.
But that's all right, it's partly deserved, and at least I get the joke.
I'm Dave Plummer, retired operating system engineer from Microsoft going back to the
MS-DOS and Windows 95 days, and today, I'll tell you about the secrets behind how that
progress dialog works, why it's so wildly wrong sometimes, and why it's such a hard
problem to get right in the first place!
We'll also test the accuracy of the shell by putting it to a real world test - copying
the entire system32 folder from one SSD to another.
And we'll do it on a PCI4 SSD raid set capable of 10Gigabytes a second, just for good measure!
Now I need to be really clear that I'm just one of many devs across the ages who has owned,
worked on, designed, or otherwise contributed to the progress dialog.
I think the very original came out of the Windows 95 project, of course, and it evolved
with Windows XP and an on down the line.
I started working on it periodically around 1995, when I added support for a number of
things that XP would have that Windows 95 did not, such as multiple streams in a single
file, compressed files, encrypted files, and so on.
At some point we added Yes to All and No to All and Skip and so on, all intended to make
life a little easier.
I worked on various aspects of it off and on until through about 2003, but only sporadically
as things came up.
The last thing I worked on, though, was in fact the time estimate dialog, so the cartoon
seemed particularly prescient.
At the time, the first, and most pressing problem was to improve the shell's time estimate
for long operations like copying a large number of files.
Even back then it was the butt of jokes at times.
Now off the cuff, you might think this is a fairly simple problem, but I can assure
you that it's anything but.
Here's the single biggest reason that the shell's estimation can be so wrong: as Raymond
Chen has aptly noted in his blog, the shell is trying to predict the future.
And it is trying to predict the future solely based on what has happened so far, such that
early in the process it's even more likely that the estimation will be wildly wrong because
it's based on only a tiny subset of the entire process.
You might think hey, I know my drive reads at 100MB second and I'm copying 500MB of files
so it should take 5 about seconds.
Experience has likely shown you, however, that it's not usually quite that simple.
A console app like xcopy doesn't provide any estimate at all, so it doesn't pre-walk the
tree of files and folders that you've selected in order to make an inventory of work to be
done.
It simply starts at the beginning and works away until it is done, at best telling you
which file it's currently working on.
But the desktop shell wants to provide much more feedback than that, so it walks the entire
selection in advance to count the files and folders and add up their size and so on.
That means the shell also knows the number of total bytes to be copied, but the problem
is that it the total size of the files is only one small factor in the outcome.
The shell does not always need to look back at the entire copy operation, but instead
could bias the calculations towards what has happened most recently.
But the reality even so is that it cannot see into the future - it doesn't know because
it can't know how much bandwidth will be free a minute from now.
It doesn't know how busy the disk or the bus will be or if writes are saturated on the
SSD or if the cache needs to be written back.
It can only assume that the future will be like the past, so it bases all its predictions
about the future on the past performance.
That sounds like a reasonable thing, but things can change along the way.
Network conditions can change, multitasking apps can slow down I/O, your multitier SSD
can slow down significantly as large copies progress, and so on.
Let's imagine something like the cartoon, where you're taking a trip from Seattle to
New York to visit a friend, and I'm the Windows Shell, and I'm trying to predict how long
it will take you to get there.
You begin your journey by walking out to the taxi.
Well, you're only going about 2 miles per hour while walking so I can assume it will
take you several months to reach New York.
Once we're in the cab on the freeway, however, we're going 60 miles per hour, so it might
be only a week or two to get to New York.
But then you're back on the little train in the terminal going 20, and the estimate gets
longer again.
Soon I update my estimate as you board the aircraft: three weeks to fly to New York.
Once you finally get underway on the aircraft, now you are exceeding 600 miles per hour.
Because the shell is smart, maybe it pays more attention to how fast you've been going
recently than at the start of the trip, but that also means that it predicts that you
will be at your friend's apartment in Manhattan about 30 seconds after landing at JFK, because
after all, you've been averaging about 95% the speed of sound for the last several hours,
and that apartment is now only a few miles away!
It's a little like that.
Just when the shell thinks it has a reasonable estimate and stakes its reputation on providing
you that guess, conditions change enough to make the prediction look downright silly sometimes.
But notice if that if you used Google Maps or Waze to calculate your trip from the airport
to your friend's house, it can be surprisingly precise, even with traffic and conditions
changing on the fly.
That's because in both cases what the computer is being asked to do is still to predict the
future.
But why can Google predict a drive uptown with far better accuracy than the shell can
a file copy from one drive to another?
It's because Google has access to vast amounts of historical data that it can draw on from
other users driving those same streets in the past.
But that model isn't very useful for computers, because the amount of time it took for you
to download and install Flight Simulator might be very different than it would be for me.
Odds are the first part of my download is more predictive than knowing your entire download,
so there's not a lot of point in using your data to estimate my download once mine has
been underway for a little bit.
You're probably better off using the current speed than the historical average from other
users on different ISPs.
For a single large copy operation, once it has been underway for a while such that the
shell has been able to gather some data about the speed of the progress, it can be refreshingly
accurate with its predictions, but this is only one type of shell operation.
Consider the case of moving a large folder, like an offline copy of the Windows folder
itself, from your internal SSD to an external data drive.
Now it's not as simple as just keeping track of how long the first part of the operation
took because measurements based on that first part might not have anything to do with the
last half of the actual operation.
What do I mean by that?
Well, let look at all the different types of files and folders you might find in a Windows
directory.
There's quite a mix - tiny .ini files, small configuration files, medium sized font files,
large dll files, and even a few huge files among them.
It depends where the operation starts.
Let's say it starts with the fonts folder, which are mostly smaller files of regular,
predictable size, usually under a megabyte.
You can tear through them fairly easily, and at a consistent rate.
So you think, well, I'll just measure the number of bytes per second of throughput that
my copy can do at the beginning and extrapolate that.
And at its most basic level, the shell is doing something just like that.
But anyone who's done a large copy and watched the performance knows that the speed can vary
dramatically throughout.
Copying large, monolithic files is a breeze, and you can crank the bytes through easily.
But when it's small files, there's lot of filesystem overhead.
For example, a file entry has to be created and populated and linked into the volume structure
in the filesystem and space allocated from the disk bitmap or whatever system it uses,
all before any bytes can even be accepted, and that's done for each and every little
file.
This really slows things down, at least relatively.
That's why copying the large dlls in system32 might breeze along at 100MB per second, yet
while copying the many fonts it drops to a mere 10MB per second.
Same disk, same filesystem.
The folders are right next to each other on the disk but copying 100 of them takes wildly
different amounts of time per megabyte.
That's why knowing the raw speed of your disk, or keeping a record of it, isn't all that
helpful.
This a problem for moving and copying, of course, but it's also a problem for operations
such as changing attributes or deleting subfolders for similar reasons because - not to beat
a dead horse here, but -- because just as you hear every time a stockbroker commercial
comes on TV: measurements of past performance are no guarantee of future results.
How can we improve the estimate?
How can me make it smarter?
Well, when I was working on it, one of the first things I realized is that different
operations can take markedly different times to complete.
Renaming files is faster than deleting files, but deleting a file is faster than moving
it when it's on the same volume than when it's on a different one.
Moving a file to a different volume is just slightly slower than copying it, because the
original must be removed and that's extra work.
But renaming files is even faster than any of these.
So, as you can see, it starts to get complicated with all the combinations and permutations.
What I wound up doing in my initial design was to break every large shell operation down
into discrete tasks, time them as they happened, and keep track of averages.
Let's take moving 100 files of 100K each from one disk to another as an example.
In our various categories of actions, we have things like creating a new file, removing
an existing file, renaming a file, deleting a file, and so on.
We also want to keep track of the time needed to move or copy any actual data as well, so
we add another category for "copying data bytes".
Then, we undertake to begin the operation.
We'll let it run for a while, display a message like 'Estimating' in place of an actual time
estimate, collect statistics, and then use those statistics to predict the future.
We'll continually feed information about the ongoing operation back into the copy engine
and it can use those statistics to update its predictions for future performance.
It will run in closed loop and continually get better.
Heck, at least near the time it approaches the end of a long operation it should be fairly
accurate.
If we're going to play junior operating system designer here, we have to treat some problems
as just practical and not theoretical - like how long do we leave up the "Estimating" or
"Calculating" message?
10 seconds?
But then you'd never get information for any operations less than 10 seconds!
What about until 10% of the data is complete, so we at least have some historical context
to guess from.
But all these rules have edge cases - like for a 4K video copy that takes an hour...
do you really want to wait more than 5 or 6 minutes for the very first estimate?
So perhaps you must make more complicated rules - 10 percent but not longer than 20
seconds, something like that.
No matter what heuristic you decide on, however, be prepared to be wrong sometimes.
And when you're wrong, people will be harsh.
Try not to be too wrong too often.
Thinking about this a little further, you could maybe store some statistics about each
disk volume in the registry.
That way, when starting off a copy from C drive, you're not working blind.
You can use some data from the last big operation on the C drive to seed the estimate for next
time.
But if last time you were copying from C to a floppy drive, does it really tell you anything
informative about copying from C to an external SSD?
Well, perhaps you're thinking you should store your prior stats based on both source and
destination.
But remember, not all networks have symmetrical performance.
And some media, whether it's a CDR blank or a RAID 6 NAS volume, writes at a wholly different
speed than you can read from it.
So, I guess you really have store those statistic by source and target pair PER DIRECTION.
And the thing is, if you were to just repeat the very same operation the disk cache may
come into play that second time around and yield completely different performance yet
again.
So perhaps you can see how trying to keep historical data, if not exactly pointless,
is much more complicated than it seems at first blush.
The copy engine and the progress dialog, not to mention the confirmation dialog, have all
evolved continually since those early days, and today, I'm curious to see just how accurate
it is.
So, let's put it to the test.
I'm going to time some actual I/O operations and see how accurate the current shell actually
is.
Let's look at a real-world example.
I'm going to copy my entire system32 folder from the boot SSD over to my data SSD, which
is actually a raid set of three fast PCI4 SSDs.
To get a sense of the raw I/O speed the source and destination drives are capable of, I'm
going to first test them both with Crystal Disk mark.
Let's do that now.
[Crystal Disk Mark Test]
Now I’ve increased the transaction size to 2 gigabytes to get around some caching
and I’ve made sure that I’ve sped this footage up as it actually takes a while to
run.
This is the boot drive and I’ve heard that there are IO problems under Windows 11 but
that seems about what I had before so I’m not sure if I’m not experiencing them or…
in any event, 2560 on the read side and we’re going to be looking at about 10 gigabytes
maximum read on the drive we’ll be writing to and a maximum write of 4200 so obviously
the bottleneck will be the source drive of 2560 megabytes per seconds and going on down
as your queue size and T count decrease.
Now that we know we can write at least a gigabyte second, and read even faster, you'd thing
we could get a reasonable estimate.
After all, we're using fast SSDs on both source and destination, so it's not like small files
are going to cause the heads to thrash around.
But if we check the size of system32, it's about six and a quarter gigabytes.
So six gig at a gig a second... carry the one...
that's like six seconds.
But we both know there's no chance it completes anywhere near that quickly.
Rather than speculate, I broke out the stopwatch and manually copied system32 so that we could
compare actual time with the estimated time.
Let's have a look!
[ System32 copy ] Alright as we watch this copy my system 32
to the data drive we can see the first estimate that comes in is one minute and 45 seconds
for an estimated 21,000 items now head stuck with his six minute 30 would have been actually
pretty close to correct but the broken clock is right twice a day too so it's hard to know
if that's luck or accuracy you're a busy person so we don't have time to watch this in real
time I had to watch it once and it was already painful watching my system 32 coffee so as
we jump ahead we'll see the next estimate runs up to 9 minutes and 30 seconds of a sudden
now at about mark we'll see this estimate gets revised all the way up to 15 minutes
oh wait I'm at 4 minutes 4 minutes becomes 7 minutes it was seven minutes you could make
a cup of coffee oh but wait it's only 30 seconds now continues to oscillate around depending
on the size and the nature of the files that it's encountering and we can see that by the
end here it says there are two minutes left at the six minute mark and it actually end
just ends so it was actually surprised by the end of the copy it appears.
As we just saw, the shell can be wildly wrong.
And I picked this example for two reasons - because I felt it represented a real-world
use case of copying a mixed payload of files, and because I knew it would be very, very
wrong.
As we saw, the transitions from a bunch of small files to a few larger files and back
and forth continually caused it to be wrong in its estimates.
Now in fairness, I don't know that the mechanism I described earlier, where it measures and
tracks historical data off past atomic events, is still in there.
Changes in I/O might have meant it needed to be completely reworked.
I have no idea.
I just know it's a hard problem and it's been solved a few different times by a few different
people, and it's still far from perfect.
But when the problem is a little more straightforward, the shell can be surprisingly fast and accurate.
We saw that it took several minutes to copy a mixed payload totalling 6 gigabytes.
But if my drive can sustain 1200 MBs/sec raw speed, why can't the shell copy files at the
full transfer rate of the drive?
Ah, but it can, and I'll show you...
Watch as I copy a single 6GB file from the data SSD to the boot SSD.
At a little better than 12000 megabytes a second, it should only take 4-5 seconds.
But what's the reality?
[ Single 6GB copy slower ]
There you go.
It happens so fast the shell isn't even done estimating yet.
But from watching the stopwatch, I can see it took right around that 4-5 second mark.
Impressive, to be honest!
What about the other way, reading from the boot drive and writing to the fast data drive?
It should in theory be even faster.
Let's give it a shot.
[ Single 6GB copy faster ]
Sure enough, this time it copies that 6GB in under 3 seconds!
It's smokin'!
So as you can see, the filesystem structure itself, and not the drive, seems to be the
majority of the overhead.
When it's just data being written to a flat file that is growing on the disk, it can come
close to saturating the maximum transfer rate of the drive.
But when there are many individual files to contend with, the operation is much, much
slower.
Particularly when measured by the number of actual bytes handled per second!
The fundamental conclusion, then, is that the shell is really bad at predicting a highly
variable and chaotic future, whereas it's much better at predicting a stable one where
the future is much like the past.
In the end, this should come as no surprise!
The other conclusion is one that is confirmed by my earlier Crystal Disk Mark results - that
manipulating lots of small files means many small I/O operations, and even if everything
else was equal, which it isn't, the drive is significantly slower when working with
small I/O blocks.
It's kind of a double whammy, and that's why it's so much slower!
Now I don't have any Patreons, and I'm not selling anything, I'm just in this for the
subs and likes.
So I'd be honored if you'd consider subscribing to my channel if you're not already.
And if you are, heck, drop a like or comment on this video so I know you're out there!
Let me know what you'd like me to feature next.
Thanks for joining me out here in the shop today.
In the meantime, and in between time, I hope to see you next time, right here in Dave's
Garage.