For Linux gurus: best way to transfer a large amount of files?

In My Humble Opinion
5

rsync It is one of the greatest tools of all time.

If it were me, I would just transfer over ethernet, because both computer would be on the network, and I ssh from one computer to another all of the time. The advantage for me is that any speed lost to ethernet is made up for in not having to mess with taking things apart. If networks and ssh are not something you’re familiar with, then copying to an attached disk may save you time.

This is a very long outline, because (looking back at the In every thread about xxxx, someone always posts yyyy), I don’t want to be the guy in a tech support thread who answers “use foo” with absolutely no explanation of how foo might help solve the problem.

Here is one way to do it. There are others. Which is best will depend on many factors.

  1. Attach the 6TB drive to your old computer
  2. Partition the 6TB drive, probably into just one big partition
  3. Put the filesystem of your choice on the drive
  4. Mount the drive on /newhome, /mnt or wherever
  5. copy the files with rsync -avPHAX /home/GESancMan /newhome/home/GESancMan
    1. a means copy all of the files, directories and stuff, preserving their permissions and owner
    2. v means be verbose about, and say what is happening
    3. P means show progress, which you definitely want for a large copy
    4. H means preserve hard links (if you don’t have any, it won’t hurt)
    5. A means preserve ACLs (again, if you don’t have them, it won’t hurt)
    6. X means preserve extended attributs (don’t have? won’t hurt)
  6. If for some reason the rsync stops, you can just start it again, and it will pickup where it left off.
  7. As often happens with me, I’ll see a bunch of files scroll by, and say “I don’t want all that stuff”. Then you can ctrl-c the rsync, go cleanup the messy files, and then restart it with rsync -avPHAX --delete /home/GESancMan /newhome/home/GESandMan which will tell rsync to delete any files in /newhome/home/GESancMan which don’t exist in /home/GESancMan.
  8. Shutdown your old computer
  9. Install the 6TB drive in your new computer
  10. During setup (or after the fact) don’t let you new computer format the 6TB drive, and tell it to mount it on /home
  11. Finish setup, or whatever, on the new computer.
  12. If the new user gesancman on the new computer has a different uid (user number) than on the old computer, then files will all be owned by the wrong user.
    1. Check using ls -l in a terminal, because it will show who owns a file.
    2. If it has your username and group, then all is good. If it has numbers or the wrong name in the 3rd or 4th column, then you have to fix it.
    3. To fix that, you can run chown -R gesancman. /home/GESancMan That might take awhile to run, because it has to change ownership of all of the files.
    4. To avoid this ownership issue, you can create your user on the new computer with the same uid and gid (group number) as on the old computer.
    5. I only know how to go these things at the command line, so
    6. On the old computer, open a terminal and run id and it should show you something like `uid=1000(gesancman) gid=100(gesancman)
    7. Create your user account on the new computer in a way that makes sure those uid and gid get used.
  13. Make sure everything works before wiping out your original computer.

As for swap, the reason you find conflicting advice is because it probably doesn’t matter. There is actually work being introduced into Linux now to allow a “hibernation only” swap, so it won’t be used for virtual memory, just for hibernating. Worst case you don’t create swap, and then decide you need it. Then you can just create a swapfile without repartitioning. The fear of swapfiles dates back to performance issues that were fixed 20 years ago or so.

I usually create swap partitions on desktops and servers because I’d rather the computer get slow than start killing things if it runs out of memory. If it gets slow, then I can go in and figure out which process (chrome) needs to die, instead of letting the kernel’s OOM killer decide. I create them on laptops, because there I do use hibernation.

Because I don’t know if Discourse has post size limit, I’m going to keep going. Here is another method of doing it, but not with as much detail.

  1. Buy a USB to SATA dock or cable for around $30 (make sure it can handle 3.5" disks)
  2. Setup the new computer with the 6TB drive mounted as /home but mostly empty
  3. Pull the /home disk out of the old computer, put it in the USB dock, and mount it on the new computer
  4. rsync everything from /oldhome/GESancMan to the your new home directory
  5. unmount and unplug the old /home disk
  6. attach and mount the game disk
  7. rsync stuff on that disk to /home/GESancMan/Games
  8. unmount and unplug the disk
  9. put the old disks in a drawer to be forgotten about