Does using the verbose mode to create a tarball slow down the actual process? I’m tarring a few thousand small files into one uncompressed blob for ease of transfer over FTP. It seems to be printing to the terminal (SSH) at about 10 files a second, but I can’t tell if this is bottlenecking the actual tar process or if that all just happens behind the scenes and the terminal printout is just slowly catching up.
If verbosity actually does slow it down, can I make tar send out any sort of “keepalive” messages, the way old FTP clients used to be able to print out ##########s every few seconds to let you know it’s still doing something?
I would be surprised if the screen I/O is impeding performance. Disk I/O is probably the culprit. Is your disk drive light pinned during the tar operation?
Technically verbose mode does slow things down a bit. After all, it does take the computer a certain amount of time to print out the verbose messages. If it is only printing out 10 lines per second though the amount of time that it is spending printing the messages is so small that it’s not worth worrying about.
I can’t tell, is the thing It’s on a remote shared server, probably virtualized, and I have no idea how much of actual physical resources it’s using. It’s not really CPU bound (no compression going on).
Well, I’m not worried about it slowing it down by 10% or whatever, I meant more “Does tar have to wait for each verbose message to be sent over SSH before it can move on to processing the next file?”
Or can it, say, finish 100 files a second, but then slowly print out the verbose files over the next two or three minutes even though the actual tarring is done?
Yes, it can. The tar process will proceed as fast as it would as if you were sitting at the actual machine itself, while sending its verbose output to your terminal session, not caring that what you see isn’t necessarily real-time. The only thing that would delay it is if it halted to prompt you for input before proceeding for whatever reason.
I used to do tons of remote work like this through extremely slow connections. If I needed the output and expected a lot of it, I’d usually just pipe the output to a file on the server instead of watching it trickle through to me, sometimes finishing much later than the process itself. Doing it that way, you can browse through, search, or just “tail” the output without necessarily having to transfer all of it over the network, plus you’ve got a nice log saved on the server if you ever need it.
Awesome, thanks! I’ll try piping it to a local file next time, that’s a good idea. I ended up using “screen” and logging off, and that worked well enough, but it’d be nice to be able to check on its progress more easily.
stdout is buffered but there are limits to the buffering. If you manage to continuously pump stuff into it faster than it can pump it back out, you’ll eventually fill up the buffer and then your program will lag.
I have no idea what the buffer size of stdout is these days.
A really nice tool for use with remote connections is mosh the mobile shell. It does almost everything you want. Apropos the OP, it will realise is is never going to be able to usefully display enormous amounts of output, so drops it. (This sounds wrong, but mosh simply attempts to keep your local screen looking like what the remote one would look like, so if stuff is flying off the top faster than it could reasonably transmit it, or be read, it is useless.)
Mosh’s real trick is managing remote connection in the face of intermittent and mobile connectivity. Don’t leave home without it.
It wasn’t all that long ago that screen output really was often a pacing issue with tools like tar.
Just a nitpick - you need to redirect it to a local file, not pipe it there.
In other words, instead of:
tar cvf myfile.tar * | output.txt
you want to use
tar cvf myfile.tar * > output.txt
Is there a reason you’re doing it uncompressed? Just add the z option when creating and extracting the tar image, and you’ll have a smaller file. That should make the ftp go quicker.
We also have no idea how old or new the Linux in use is.
Hmm, interesting. Mosh looks really neat, and it says it’s not a daemon, but there’s at least a server executable. Gotta see if that’ll run on this virtual host.
Doh, yeah, I used the wrong term
They were a bunch of JPEGs and PNGs. GZ won’t further compress them much; it’d just waste CPU cycles on both ends.
I’m not really even sure if it’s Linux. Just some sort of *nix with tar GNUAnonOs.
It’s on a remote shared server, probably virtualized, and I have no idea how much of actual physical resources it’s using. It’s not really CPU bound (no compression going on).