I have a customer who needs to make offsite backups over the interent.
He has a small server in another location that has plenty of disk space for a month’s worth of backups from his main site. He has a DSL line at both locations, giving him about 640 kbps up and down from both offices.
So we want to set up an automated backup of the server at one office, and have it backup to the server at the other office, using compressed files, a secure connection of some kind, encryption, etc.
A full backup is about 10 gig, which they want to do weekly. A daily incremental backup is about 1 gig. They run six days a week, so the full backup has to be done in 24 hours or so max, so it can start Saturday night and be done by Monday morning. The incremental should take no more than a couple of hours.
Any suggestions for software to automate this? We’ve found a couple of candidates in the $800 range that seem to do the job, but since I have no experience with them or the companies that make them, I don’t have a good yardstick to judge them by.
I was hoping someone out there might have some experience and be able to point us a reliable, easy to use solution that they have seen or used and can recommend.
DSL is rarely symmetrical: you’ll need to check this. You may also need to consider bandwidth limits (so many GB/day etc). On a 640Kbps fully symmetric link, 1 GB will take about four and a half hours, assuming full speed - DSL is often shared bandwidth. So that’s 45 hours for 10 GB - not enough bandwidth for your weekend backup. You need a faster link. Be prepared to pay. However, your ISPs might support a burst traffic mode - you normally run at speed X, but for a short time you can run at speed Y (usually during the very quiet times, like 02:00 - 05:00).
The link is relatively easy: a VPN. But some ISPs forbid or block this.
I’ve checked the DSL, it is symmetrical. I’ve checked with the ISP (same at both locations) and they everything is cool with them, we will not exceed any bandwidth limits, and the bandwidth is not shared (I’ve never actually seen shared bandwidth on DSL, although I’ve seen lots of cable connections that were shared). They also say that VPN connections are not blocked. I checked all of that first thing when this came up.
I’ve already set up a VPN link between the systems for testing. There is a software firewall issue on one end that complicates it, but it could be done if it has to be.
I’m not sure of your math on the speed. According to my back of the envelope calculations it should be more like 3.5 hours per gig uncompressed. This seems to be born out by my tests, which take about 12 hours to transfer 8 gig after compression, with an average of a little better than 50% compression (around 3.8 gig actually transferred). This makes the full backup very doable over Saturday night and Sunday.
So I know it can be done in this specific situation, I’m just hoping someone has some experience with some specific software so I can recommend something to my customer using something beyond “well this looks good in the product description”.
Really? DSL connections are almost always set up in groups to share the bandwidth; they call it ‘contention’ - a good contention ratio would be 20 to 1 (20 people on one pipe). Then again, DSL is also usually assymmetric, so it’s clear you don’t have a run-of-the-mill setup.
I concur - it’s coming out at 3.64 hours for me (although it should be noted that I’m not famed for my maths), however, my calculations were based on full theoretical throughput of bits with absolutely no consideration for protocol overhead (i.e. the number of bits you use just encapsulating the data in packets, and sending commands/acknowledgments, etc to the remote machine), lost packets, etc.
Hmm. In both places I’ve lived in the DSL era, NOT sharing the bandwidth has always been one of the advertised benefits of DSL over cable modem. None of the three DSL connections I’ve had in the last few years (one symmetric, two asymmetric) has been shared: performance of “the Internet” as a whole might drop during “peak times,” but my connection speed to my ISP is always the full rated speed - never faster or slower.
How would you be measuring/assessing your connection speed in these cases? I’m not disputing what you say (indeed, if your connection is contended, it should say so somewhere in your terms and conditions), just curious as to what you mean about connection speed.
I don’t know about any software to do this, but our company uses an outside vendor - Connected Corporation (http://connected.com) to do this. The backup is daily, it’s compressed, and it is fairly painless after the first time. I have no idea of the cost of this service or if it fits your client’s needs, but it does seem to work quite well, even at DSL speeds.
Most ISP’s have a “check your connection speed” dohicky that does a quick upload or download. If not, SpeakEasy, DSL Reports, and various others have connection speed testers (admittedly testing more than just the connection to your ISP)…
…but I never used those: I just uploaded or downloaded a large, compressed file from the hosted site at my ISP and timed it or watched the connection indicator.
Re: the OP: I do backup of my Mac to iDisk over 1.5 Mbps DSL all the time, but it’s a tiny chunk of data most days, so I can’t really speak to the practicality of it. I don’t do it for larger data set backups: I burn them to DVD and carry them offsite.
Actually, I’ll amend what I said a little and kind of go with what TimeWinder said. Of the about 8 different DSL connections I’ve paid for, and the ones for customers/family/friends where I’ve asked, the ISP has said that they don’t share bandwidth.
But things are sometimes slower than other times. I’ve always assumed that the slowdown didn’t represent the speed from me to the ISP, but as you indicate, I have no good way to measure it either. So maybe they are shared at some level, and the ISPs are either lying, mistaken, or getting us on some technicality. But in the cases where I’ve asked directly, I’ve always been told that the DSL line in question was not shared.
As for the SDSL vs ADSL, it seems pretty variable. Around here, and up to about the 640 kbps speed range, symmetric seems pretty common with “business” packages (which also usually include static IPs and better service guarantees, etc.) and ADSL is more common with “home” type connections. And I have noticed that almost no one seems to offer a symmetric at over 1 meg (or even 640k), and if they do, it really runs the price up. But 640 kbps symmetric is not really uncommon around here, at least for businesses.
It may just be that the typical DSL offering is different here in the UK to what you have in the USA; I’ve found this to be the case with a number of other things. certainly here, contended bandwidth is the norm - not that you’d notice, because it always just seems to work at the speed it should.
Interesting, I never thought of doing this. It does seem like this would be a good way to gauge your connection speed to your ISP. We so rarely use any space that always comes with an account at an ISP because we (and my customers) tend to run our own servers and host our own content, either on site or at a co-location. So I never even thought about testing against the ISP’s servers.
I’ll have to try it a few times and see what kind of results I get.
It just goes to show, sometimes it the simple answers that you don’t think of.
I’ve looked at the connected.com website and it looks like that is a possibility, although everything I see indicates their process backs up to their servers, not to a server of your own. I’m waiting for some pricing information from them and then I’ll show it to my customer.
But if anyone has recommendations for software you may have used for backup that can target a backup to another machine across the internet, I’d love to hear them.
Depending on the OS environments involved and cost constraints, rsync might be a good solution. It only copies changed files to save bandwidth, and can incorporate SSH encryption. It’s a standard tool in any Linux distribution, but can run on a Windows box with a little extra work.
Here’s a fairly detailed link on the subject - a few minutes more searching would find you a lot more I’m sure.
Thanks, I’ll look at it. I’m about a minus 1 one the scale of *nix geeks however, so I’m probably not going to be able to do much with it.
We’ve located a program call Rbackup from Remote Backup Systems, and have installed a demo. It has the features we want, but seems slightly flakey, although that may be my ignorance in setting it up.
It runs about $800, and that is not out of line for this customer, so if anyone has any suggestions in that range, send ‘em over.
What type of backups do you need? Do you need “ghost” drive backups in case of total system failure, or just critical files duplicated across multiple sites / servers? If the latter, there are many a sync program out there that are very reliable and reasonable. There’s a list of file mirror / sync software providers here:
I’ve used SmartSync with good results, although only over a local network, not over an SSH tunnel on the internet.
One other consideration is that both SSH or VPN tunneling requires the data to be encrypted… so transmitting 1GB of data will actually be some higher number, because of the overhead of the encryption…
We’re looking just to get an offsite backup of about a dozen folders that total up to 8 or 10 gig. These are on a separate drive from the OS. The data ranges from text files to HTTP source to Quickbooks data to data in a proprietary format used by a custom dispatching system we wrote for them. A little bit of this is SQL data (we run Pervasive SQL manager) but most of it a mixed bag.