I’m copying a rather large amount of data (10,000 files, about 38 GB) from my hard drive to a flash drive. It starts by telling me it is copying the data at about 9.8 MB per second, and the job will take about an hour. Soon that was down to about 6 MB/s, and it told me 7 hours. At the moment it’s telling me the flow rate is 4.9 MB, and the time estimate is 13 hours.
Aside from the question of why the speed is varying so much, it seems these estimates bear no relationship to each other. If the data rate slows by 30%, why would the time estimate go up by a factor of 7? And another 20% decrease nearly doubles the time estimate yet again.
Where are these numbers coming from?
Also, why does the speed vary so much, and always seem to trend downward? I know my computer does other stuff in the background tha what I tell it to do, but other than writing this post I haven’t burdened it with any other work. Yet the speed is down by half of what it was when I started. That is – it’s down by half if either posted speed was accurate, or at least calculated consistently, which is questionable.
Oh, one other thing. I defragged my disk before starting in the hopes that it might speed the process up – not having to jump all around the drive and all. All this data is contained within one folder and a set of subfolders.
FYI, the rate was pretty consistent at about 4.7 MB for about 20 minutes, and the job estimate was steady at 18 hours. Now the speed is almost unchanged, but the total time estimate is down to 6 hours. Plus it says there remains about 18 GB to copy, which means it is more than half way done. and of course it’s only been running for about an hour since the start (I should’ve checked the start time more carefully).
I might be pulling this out of my ass, but I believe the time estimates are based on some kind of averaging. You often see the speed fluctuate wildly while the estimated times remain within a narrow range.
Now, I’ve observed the above behavior with downloads, but usually my file transfers run at a pretty constant speed determined by the slowest connection along the line between origin and destination.
Were you doing anything else with your PC while these files were being copied?
Is it possible that you were running one or more other apps that were copying files and that slowed things down?
Did you consider posting your question on a Windows Help board?
This is a fine board with many knowledgeable people. But for serious questions about Windows, I find that most really expert people belong to forums like:
If I was actually uploading or downloading anything, the speed variations would make a lot more sense.
Like I said, only running Google Chrome for these posts.
Also, I went into the Task Manager, and ended a few processes that I was pretty sure weren’t critical to the copying, and the speed didn’t change one bit.
And for the record, now it’s saying the speed is 4.4, there’s less than 12 GB left to copy, and the time estimate is back up to 19 hours.
And just in case anyone was wondering, this is Windows 7 Home Premium running.
The flash drive is probably compressing the data by first consuming large chunks which take time to compress without increasing the amount of data transferred as seen by Windows. So when the averaging occurs, there could be 2 large writes to the flash drive with 1 long wait in between, or 2 long waits with 1 large write in between, or any combination of large writes and long waits including partials. This throws the averaging algorithm all out of whack. You can check the flash drive’s website to see if there are faster transfer modes. Those would usually decrease storage density on the drive.
No factual answer, but I’ve noticed speed seems to vary with the size of the files. If I remember, copying lots of small files is slower than copying large files.
Windows 7 also seems to group the files by size. So lots of small files will copy, then medium files, then at the end really big files. Each big file takes a long time to copy, but the overall speed picks up considerably.
How would a flash drive be doing any of this stuff without Windows knowing it? It doesn’t have any software of its own, Windows is controlling every step of the process.
I’ve noticed the same thing. My guess has been that Windows’ estimated completion time is based purely on the file size, and it ignores the fact that for each file, in addition to copying the data in the file itself, it also has to make a directory entry for the file name and properties. So when it is working on a gazillion tiny files, that throws the whole estimate off.
Handling a huge number of files in copying or moving has always caused windows to slow to a crawl. This has been true since windows 98 and it persists today. I’m not sure why, but it has all the appearance of a buffer being overloaded on some level. That this behavior would persist across all these versions and years says to me it has to do with some pretty fundamental drive IO & software interface issues.
If you have the patience copying files a few hundred at a time vs thousands at once will often get the copy job done much faster. It obviously requires more hands on though.
The flash drive has drivers provided by the manufacturer. Windows only sees a virtual view of the flash drive as a disk, and doesn’t know about the compression techniques it is using.
ETA: On top of the other problems mentioned above, Windows doesn’t know ahead of time how scattered the files and their blocks are on the disk and can’t predict how much head movement is required. All of these factors are exacerbated when using a flash drive. But it’s just poor estimation. The data is being transferred as fast as it can whether the progress bar was there or not.
This isn’t exactly a new observation. The obligatory XKCD.
The algorithm used seems to use a moving average which uses a very poorly chosen (too small) window. So simple purtubations in the progress (for instance due to disk seeking, operating system activity) are amplified.
One of the little things I like about Mac is that it doesn’t pretend that its estimates for things like this are precise: It’ll say things like “About five minutes remaining”, or “less than a minute”.
I agree. And prior to starting the copy, Windows has no idea how long it will really take. I supposed that it could scan the entire folder and come up with a more accurate estimate of total copy time. But I doubt most people want to wait for something like that.
I find it interesting that Microsoft chose to display an inaccurate time estimate rather than display the percentage complete which is reasonable feedback for a job you simply can’t know ahead of time how long it will take.
This. When I was first out of college circa 1990 I had a job writing install scripts for a big software company. You would not believe how much angst and effort went into the algorithms for developing progress bars. I attended a conference where one of the sessions dealt solely with this topic, and hundreds (dozens ?) of us from across the country compared notes. I’m not sure the science has advanced much since then. it’s simply based on calculating how much there is left to be done and the rate at which it has been done so far. Some algorithms calculate base on number of files, some on size of files, etc., so it varies.
The reason why the calculation changes is that the speed changes. Just like when you’re driving a car… if you’re heading to the store a mile away and you leave your driveway at 30 mph, you’re going to get an estimate of 2 minutes. Then after you reach the corner and spend a minute at the stop light, the estimate is going to change dramatically.
This sort of variation is a significant issue when copying files over a network (as opposed to what the OP is talking about). But fragmentation can also be an issue.
If I was actually uploading or downloading anything, the speed variations would make a lot more sense.