For Linux gurus: best way to transfer a large amount of files?

In My Humble Opinion
1

I’ve got a new computer on the way. My current system is six years old. When I bought it, I was still using Windows; I made the switch over to Linux (specifically Kubuntu) about four years ago. I’ve learned a lot about Linux, but I am by no means an expert.

I’m not even going to give the new computer a chance to boot into Windows. The first time I turn it on, I will boot it directly to a Kubuntu stick, wipe out the current partitions, and start from scratch.

The hard drives in my current computer are even older, I’m guessing eight or nine years. They’re showing no signs of slowing down, but I’m taking advantage of this time to replace them - better safe than sorry.

First of all, I want to be sure my understanding is correct:

In my current setup, there are three hard drives. sda is the boot, a 128GB SSD. sdb and sdc are each 2TB SATA drives. sdb is my /home partition, and all of my games are on sdc. I’ve got sdc mounted as /home/GESancMan/Games. Thus, when I do ‘ls’ on /home, Games shows up as a directory in the list.

I bought a 6GB SATA drive to replace sdb and sdc. My plan is to copy everything on both drives over to the new one. I’m going to copy everything from sdb directly over, then mkdir Games on the new drive, and copy everything on sdc into this new Games directory.

The end result should be that my directory structure will be exactly the same, but everything will be on one hard drive. And since on the old system my games were located in /home/GESancMan/Games, and on the new system my games will be located in /home/GESancMan/Games, everything should run as before; as far as the OS is concerned, nothing has changed. Do I have all of this right?

My second question is on how best to perform my copy operation. I could do it over my network, or even hook the two computers up with an ethernet cable and copy directly. I’ve Googled how to do these things, and it looks like it would be a big pain in the butt trying to figure it out. And I suspect the transfer speed would be slow.

The new machine has a 512GB SSD, which I plan to use as the boot drive, as before. Then my 6TB drive will be sdb on the new machine, and that is it. So I’m thinking the best way to go would be to install the 6TB drive in my old computer, mounting is as /newhome, copy everything over, then install it in the new machine. Then when I install Kubuntu on the new machine, I can simply select the 6TB drive as my /home partition, and be done. Does this sound like a good plan?

2

Oh, and a follow-up question: The new system has 32GB of RAM. Do I even need a swap file or swap partition? Lots of conflicting info out there on this issue. I don’t use hibernation, so if that’s the only reason I might need a swap, I’d rather just go without.

3

How fast is your SSD? It’s true that copying 4 TB of data will take all day if you use something like gigabit Ethernet, so you could save a lot of time not using the network.

Last I checked, the Ubuntu installer has an option (under “Manual Partitioning”) to use an existing partition and not re-format it, so that should work if you have all the files pre-copied on there; not sure what is your question about it?

If you do not need to hibernate then you do not need a separate swap partition.

4

Right, that’s the plan. I just want to be sure it’s the best plan. I’m pretty sure it is, but if there’s a better way I’d love to hear about it. :slight_smile:

Do I need a swap file at all?

5

rsync It is one of the greatest tools of all time.

If it were me, I would just transfer over ethernet, because both computer would be on the network, and I ssh from one computer to another all of the time. The advantage for me is that any speed lost to ethernet is made up for in not having to mess with taking things apart. If networks and ssh are not something you’re familiar with, then copying to an attached disk may save you time.

This is a very long outline, because (looking back at the In every thread about xxxx, someone always posts yyyy), I don’t want to be the guy in a tech support thread who answers “use foo” with absolutely no explanation of how foo might help solve the problem.

Here is one way to do it. There are others. Which is best will depend on many factors.

  1. Attach the 6TB drive to your old computer
  2. Partition the 6TB drive, probably into just one big partition
  3. Put the filesystem of your choice on the drive
  4. Mount the drive on /newhome, /mnt or wherever
  5. copy the files with rsync -avPHAX /home/GESancMan /newhome/home/GESancMan
    1. a means copy all of the files, directories and stuff, preserving their permissions and owner
    2. v means be verbose about, and say what is happening
    3. P means show progress, which you definitely want for a large copy
    4. H means preserve hard links (if you don’t have any, it won’t hurt)
    5. A means preserve ACLs (again, if you don’t have them, it won’t hurt)
    6. X means preserve extended attributs (don’t have? won’t hurt)
  6. If for some reason the rsync stops, you can just start it again, and it will pickup where it left off.
  7. As often happens with me, I’ll see a bunch of files scroll by, and say “I don’t want all that stuff”. Then you can ctrl-c the rsync, go cleanup the messy files, and then restart it with rsync -avPHAX --delete /home/GESancMan /newhome/home/GESandMan which will tell rsync to delete any files in /newhome/home/GESancMan which don’t exist in /home/GESancMan.
  8. Shutdown your old computer
  9. Install the 6TB drive in your new computer
  10. During setup (or after the fact) don’t let you new computer format the 6TB drive, and tell it to mount it on /home
  11. Finish setup, or whatever, on the new computer.
  12. If the new user gesancman on the new computer has a different uid (user number) than on the old computer, then files will all be owned by the wrong user.
    1. Check using ls -l in a terminal, because it will show who owns a file.
    2. If it has your username and group, then all is good. If it has numbers or the wrong name in the 3rd or 4th column, then you have to fix it.
    3. To fix that, you can run chown -R gesancman. /home/GESancMan That might take awhile to run, because it has to change ownership of all of the files.
    4. To avoid this ownership issue, you can create your user on the new computer with the same uid and gid (group number) as on the old computer.
    5. I only know how to go these things at the command line, so
    6. On the old computer, open a terminal and run id and it should show you something like `uid=1000(gesancman) gid=100(gesancman)
    7. Create your user account on the new computer in a way that makes sure those uid and gid get used.
  13. Make sure everything works before wiping out your original computer.

As for swap, the reason you find conflicting advice is because it probably doesn’t matter. There is actually work being introduced into Linux now to allow a “hibernation only” swap, so it won’t be used for virtual memory, just for hibernating. Worst case you don’t create swap, and then decide you need it. Then you can just create a swapfile without repartitioning. The fear of swapfiles dates back to performance issues that were fixed 20 years ago or so.

I usually create swap partitions on desktops and servers because I’d rather the computer get slow than start killing things if it runs out of memory. If it gets slow, then I can go in and figure out which process (chrome) needs to die, instead of letting the kernel’s OOM killer decide. I create them on laptops, because there I do use hibernation.

Because I don’t know if Discourse has post size limit, I’m going to keep going. Here is another method of doing it, but not with as much detail.

  1. Buy a USB to SATA dock or cable for around $30 (make sure it can handle 3.5" disks)
  2. Setup the new computer with the 6TB drive mounted as /home but mostly empty
  3. Pull the /home disk out of the old computer, put it in the USB dock, and mount it on the new computer
  4. rsync everything from /oldhome/GESancMan to the your new home directory
  5. unmount and unplug the old /home disk
  6. attach and mount the game disk
  7. rsync stuff on that disk to /home/GESancMan/Games
  8. unmount and unplug the disk
  9. put the old disks in a drawer to be forgotten about
6

That is awesome, thank you so much. I’ve been going back and forth on whether to go with my plan above, or to set up the new computer first, then copy stuff to the 6TB drive. The latter would allow me to be a little more selective about what I copy, without feeling rushed, because I’m sure there is junk in there that I no longer need. I wouldn’t have thought of the USB to SATA method!

7

What model of 6TB :slight_smile: SATA did you get? If it’s shingled, the transfer will slow down and take a long time to complete.

8

I made sure it’s CMR. Seagate Ironwolf.