How to copy a large number of files without corrupted files derailing process?

Factual Questions
1

I’m retiring very soon! And one of the few remaining tasks is to copy all of my work computer files to a shared networked location (already designated). The trouble is, in the past whenever I have tried to do this by selecting all and choosing copy in Microsoft FileExplorer, occasional corrupted files will keep halting the process. A small number of the files are corrupted and I just want to ignore their loss. But, since I can’t control and don’t know how the system is choosing the order in which to copy them, I don’t have any practical way to respond to each halt by deleting the corrupted one and then selecting the files that weren’t copied yet and asking for just them to be copied. I think the number of files that will turn out to be corrupted, and the total job, are big enough that just deleting that corrupted file and starting everything over from the beginning each time may not be practical.

I have 400 GB to copy, 200,000 files in 6000 folders, all in one top level folder. I work remotely. We have Microsoft OneDrive and all the work files I need to copy are under its control. I normally access them through a logical drive mapping to my local laptop PC’s A: drive, but I can access them through the official OneDrive URL (which is how I’m supposed to do it), if whatever software I’m using to access them can handle the spaces, uppercase/lowercase, punctuation, and 40 characters OneDrive adds to the path and file name. IIRC, when we started using OneDrive, I started finding files whose total path and name were too long to be accessed. I keep some of the files available as local copies on the laptop, but not the great majority of them. I’m not allowed to install software without an approval process that takes longer than I expect to be working, so doing this with some new software product isn’t an option.

Once upon a time I went to our IT department because I needed a fixed IP address on our company network, and they told me to just start pinging addresses until I found one that didn’t answer, and park myself on that. I think this is a terrible idea! And it’s just one of the reasons I’ve never trusted their advice. Another is the fact that, with IT’s help during an earlier computer transition, I permanently lost all my computer files from my first 17 years here, with no backup working. Gripe gripe gripe! I suppose I will ask them about this copying job, which has the advantage that if something blows up, the company will own the issue, and I will have fulfilled my commitment; but I hope I don’t have to rely just on what they say.

I have the suspicion that this task is already done, in a sense, because the fact that they’re all already part of the OneDrive system means that they are already copied to a shared networked location. However, IT and leadership have already created a designated place for me to copy to, and told me that the copying is a task I must perform, so in another sense I know the task is not already done. I think the fact that I judge it necessary to proceed by posting this question here on the Dope speaks volumes.

Thanks for any help!!

2

Oh. Work computer.

So you can’t install software to do this for you, huh? Do you have an IT Department that could okay the installation of useful software? Rereads oh. Ugh.

ETA: and this is your problem and not the responsibility of the organization?

3

No chance of being able to run a chkdsk and fix/remove the corrupt file before copying?

4

Do you have the time (and patience) to monitor the copying process?

When I was facing tasks like this, I tried to break the job into multiple, smaller tasks. You state that you have 6000 folders to copy, each containing an average of about 33-34 files and 15 GB. I would sort the folders alphabetically, select the first 20 or so, and copy those. When that job finishes, select the next 20, and copy those folders. And so on. When a corrupted file interrupts the process, you will have a much easier time of deleting that file and beginning the copy process again, this time starting with the folder which contained the corrupted file.

Yes, this will take a lot of time and it’s a tedious job of performing 300 copy actions. But you will at least have some control over the order in which the folder/files are copied.

5

You could use xcopy (in a cmd window) and use the /C switch to ignore
errors.

6

Use rsync or robocopy

7

I think the right tool for the job might be rclone. That does not mean it is an easy tool to use, or simple to setup. (A backhoe might be the right tool to dig a large trench, doesn’t mean you or I know how to drive one!)

rclone is a command line utility used to copy files from cloud storage (such as OneDrive) to someplace else. That other place can be a different cloud storage provider or a local drive.

For windows, it can be a single executable, so there is nothing to “install”, but if your work computer is locked down in such a way that it won’t run anything except the allowed software, then it won’t work there. The thing with rclone, is you don’t have to run it on your work computer.

It sounds like the goal is to get your files out of OneDrive and onto a corporate file server. Are you connecting with a VPN and then mounting a location on the file server?

First step after copying the rclone executable to your computer will be to add OneDrive as a location. The directions for that are at Microsoft OneDrive Open a cmd window, go to where the rclone.exe file is located, and run rclone config. Then answer the questions, and at the appropriate time tell OneDrive that rclone has permission for access. Once that is all working rclone lsd onedrive: (or whatever you named it) should show your folders on OneDrive.

The next part is to download your files from OneDrive, and upload them to the network share. You can do that in one or multiple steps.

In one step it will look something like rclone sync -P onedrive: \network\share\mount\point

A multi step process could be something like
rclone sync -P onedrive: \external\disk
Then copy from \external\disk to your network share using FileExplorer.

rclone sync should continue on errors, so if any files can’t be downloaded, it will just keep going and grab the rest.

8

I use an app called Roadkil’s Unstoppable Copier in situations like this. Very effective. Not sure if the newer versions are as good as the version I downloaded many years ago and have not updated.

9

The command-line tool Robocopy is built in to Windows and is almost certainly the path of least effort and highest reliability for our OP assuming his work machine is a Windows machine.

I’m sure the OP wants to do a workmanlike job, but also isn’t interested in sweating the very last byte to perfection.

Fire up a command window, practice a bit with Robocopy until you’re confident of your settings, then turn it loose await the results. It can also log all that it does so with a bit of searching and sorting in the log file one can find all the glitches and remedy the ones that are a) important enough to bother with, and b) remediable in the first place.

10

I like what I’m hearing.

It looks to me like the best thing to try is:

robocopy a:*.* X:\EvilEmpire\PatheticVictimAttempts /e /z

and I’m about to try it with a small subset of . (which I’ll delete later).

Interesting tidbit: LSLGuy, 15 years ago you were kind enough to write a small Windows program that adds a mouse menu option to make File Explorer copy the path and name of the highlighted file to the clipboard. I used that for years! Something blew it up eventually, but I still have it. And it’s one of many plums that are getting swept up in this tornado.

Though, our IT department somehow limited us from moving files with an .exe extension around, so at first I couldn’t make it work. They said it was to prevent us from bringing a virus into the network. So I changed the extension from .exe to .virus and that worked fine, I could send and move it everywhere.

11

FWIW this functionality exists in Win10 File Explorer if you + on the file name and choose "Copy as path).

12

Hmm. There are spaces in the path to the destination folder, and robocopy considers the next word in this path to be Invalid Parameter #3. It seems to interpret spaces as delimiters for paths. Does it not accept spaces in folder specifications? The spaces occur earlier in the path than I have control over.

Trying it with quotes around the destination and that seems to be working. But gotta run…

13

Always, and I do mean always put quotes around paths in command line tools.

And yeah, the Explorer “copy path to clipboard” feature was added in the next version of Windows after I whipped up that tool.

14

So, “+ on the file name” means shift-right-click?

15

Can you zip the files, copy/move to your destination, then unzip? I learned long ago that it’s faster and easier to zip a large number of small files and copy one zip file than it is to copy each file individually.

16

So noted. It’s working now. I see it doesn’t copy the folder, but, rather, its contents, so that folder stops appearing as wrapper around those contents.

And in what language is “robo” short for “robust”? That just dawned on me as I was driving to physical therapy. No wonder I need to retire.

17

Agreed - that is terrible advice - I actually had to track down and deal with an IP address conflict that came into existence pretty much that way - someone assigned themself a static IP that happened to belong to a printer that wasn’t switched on at the time they did their ping. I was trying to troubleshoot the reported issue (printer doesn’t work sometimes) and was confounded for a while by the fact that ‘it’s responding to ping, so not a network issue…’ (it wasn’t the printer responding)

18

The language where max 8-character names for apps were still required and the film RoboCop was released the same year. That language :slight_smile: .

That depends hugely on the speeds of the various networking components involved.

If he’s moving the data to a NAS, the act of unzipping on the NAS means streaming all the zip data back to his PC to perform the unzip there then send each file back to the NAS individually at full size. All that after first zipping, and then copying the entire zip as a unit onto the NAS. That would be monumentally inefficient.

Now if his use case allowed him to just leave the zip closed up as-is on the NAS there’d almost certainly be benefit to zipping then copying the zip. At least for him. If anybody wanted to later use that zip, they’d be the ones paying all the extra to get one or many of the files out.

Of course depending on what the files contain, they may already be fully compacted and so a zip of them would be larger than the files themselves.

A lot of the seemingly eternal verities of the Windows 3.1 or even Windows XP era are simply no longer true. Or at least not necessarily true depending on a host of factors that hadn’t been invented yet way back in Ye Olde Darke Tymes.

19

I’ve never used robocopy as the last time I needed to do such things predated its existence (back in the days of Windows XP). I note that (a) it will not copy files which are being held open by an app, and (b) it doesn’t seem to have an explicit option to ignore copy errors (like “/c” in xcopy). So what happens when it runs across files it can’t copy? I assume from the “robust” claim that it just logs the failure and carries on, correct?

I also note that there are many backup programs that can copy pretty much anything including open files and active system files, such that you can back up the system drive while the computer is fully in use. I’ve never felt 100% comfortable with that approach but it does seem to work. Pretty much the only thing not backed up is the system page file which is obviously not necessary.

20

A job this important requires the right tools.

IT should be tasked to use backup software. Sync it until the OP’s retirement. The backup should reflect the last work done.