NAS & backups -- best advice?

Does anyone have any experience with NAS (Network Attached Storage) for small businesses? I’d like to get opinions on what is out there for those with storage needs in the 10-100TB range; reliability, cost, gotchas.

I am presently working with two Drobos; an 8-bay and a 5. The 5 bay presently has 5 3TB drives and I am gradually replacing the 3s with 4s. Each one takes 48 hours to reconfigure, and although I can still access the drive, it’s pretty scary to have all my eggbytes in one basket.

Both Drobos occasionally disappear from the drive list on the network, and their tech support hasn’t been able to find out why.

The firmware in the Drobos seems less than ideal. Some common functions are frustratingly slow, like deletions.

On a related note, any opinions about industrial-strength backup programs? Drobo has discontinued their DroboCopy option. I need something to do unattended, scheduled backups from unit 1 to unit 2, incrementally, on a low-priority basis so it doesn’t impact other work in the foreground. A terrific option would be to transfer data directly, peer to peer, from one unit to another, freeing up the PC’s CPU and resulting in one-half the amount of data going over the network.

And, no, using the cloud is not an option for storage or backup. There’s too much data to conveniently move thru a relatively slow Internet connection and the cost is prohibitive.

Can you describe your needs a little better ? Why so much data ? Are you running large databases or virtual machines off it, or is it mostly some sort of archive that is rarely accessed (ie, say, scanned documents for the last 50 years from an insurance company). Do you have a tape drive for offsite backups ?
I know qnap and synology both make 8 bay NAS. But as a software developer if I were doing this for myself I would probably just build out my own linux box using any of the many 8 hot swap bay/2u+ rackmount cases, some random single xeon motherboard, and an add in RAID card (probably the most expensive part after the drives). As ultimately what you want is pretty simple to do in linux from the command line (samba for file sharing, rsync for backups, or something based on rsync like duplicity)

Most data is original video, now moving into HD. The needs are (primarily) long term storage for archive purposes. Tape units these days are less attractive than disk (not sequential, too esoteric, may be obsolete sooner, cost more per byte, incompatible with other access). The intention is to store as much as possible for as long as possible (perpetually, although migration will undoubtedly be necessary).

Since the cost per byte has been declining for at least 70 years, perpetually isn’t all that impractical. As the data grows, the cost declines.

Off-site is provided by taking one machine to another location periodically, then swapping, so copy ops are needed to sync. Maybe in the future we’ll get a third box so no two are at the same location at the same time in case of an accident. If fiber speeds get faster and cheaper, that will change the plan considerably but I don’t see that happening for 5-10 years.

While I’m not adverse to building a Linux box from scratch (I’ve built several hundred PCs from scratch), I’d prefer a turnkey solution if available. The days are long gone where you have to reinvent a wheel for simple tasks.

The technical side is trivial: you have a set of boxes with sufficient capacity and you copy your data to them and you rotate the boxes off-site. Or, if you have sufficient bandwidth, you hire space in a data centre (e.g. Amazon Web Service) and periodically sync.

What you need to establish first is the business side: what do you want to do? Key issues to consider include: the legal concerns, affect of data loss (slight, major, and total) on the business, security of the backup. And, of course, budget.

If you go for cloud-based backup, be sure to have separate off-site backups of encryption keys, account information, etc. And to kick off the cloud-based backup, they’ll typically want a set of hard drives for the initial upload.

Well synology supports backing up to another synology or any rsync target:

http://www.synology.com/support/tutorials_show.php?q_id=461

Buffalo Terastation supports backing up only to another terastation, or a directly connected usb drive

(http://www.smallnetbuilder.com/nas/nas-basics/30771-backing-up-your-nas-harder-than-it-should-be)

The problem with most of the off the shelf NAS boxes is that the processor is kind of weak sauce and they could use more RAM (although many are upgradeable). That can be better for power usage but doesn’t help the speed any. You hardly need to reinvent the wheel with anything in linux today - there are probably half a dozen NAS distros like FreeNAS that require no more config than an off the shelf NAS.

I don’t have any tape drives in service right now, but looking at pricing after the initial sticker shock for the drive itself the tapes themselves are maybe 1/2 the cost of similar size HDDs, but your point about the other issues with tapes is spot on.

I’m pretty sure I answered those questions already, and all those items have been considered.

Now do you have any advice as to the hardware I’m looking for?

I’ve been through several generations and configurations of tape drives. At one time, the ratio of disk to tape cost was so great that tape made sense. That ratio has changed drastically; tape costs have not come down much, but disk costs have come down radically in the last 15 years.

And while I don’t absolutely have to have 100% of my archive data available at disk speeds, the specter of having to dig out and transfer gobs of data from tapes just to do a feature compilation video from multiple sources is just too time consuming to even anticipate.

I just did a compilation video from 10 years of New Year’s Eve Polar Bears, and in order to edit 20 hours of video down to 10 minutes, I had to have 150GB online all at once. And the storage for HD video will soon balloon to 4 times the standard video data.