I have huge files ~4TB a piece for work that I need to handle. What is the largest commercially available hard drive I can get (I am thinking 16TB but the largest I found was 12TB) for under $1000 that will function as an external drive?
Can a normal HDD enclosure handle a 16TB drive?
What are the pros and cons of a single drive vs. RAID (not for multiple copies just more space)?
Originally RAID was meant for your situations similar to yours - Redundant Array of Inexpensive Disks, so I guess RAID 0 would solve your problems with no significant downsides. I don’t know that much about computers so I can’t really give a much more in depth answer than that.
You can get a USB-attached RAID array and set it up as RAID0. It will appear to your OS as a single gigantic disk. That will probably be cheaper than buying top-of-the-line massive disks individually.
Very partial answer:
One drive: if it dies, your files are gone. A bad thing if they are more than just temporary scratch files.
RAID: a bit more expensive, but redundant. E.g., you buy 3 drives, and can tolerate a failure of one of them.
Multiple drives = more storage, obviously, but take into account the RAID configuration, so 3 drives but only 2x storage in the above example.
Cost: instead of buying the biggest available drive, maybe buy a couple of the second-biggest if it’s significantly cheaper. So 2 x 12 TB instead of 16 TB?
External connectors: I’m not an expert here; presumably eSATA and USB 3 are available.
Don’t use RAID-0. That’s a terrible idea. Use RAID-6 (or at least RAID-5 but RAID-6 is better since it is more likely to recover your data in case of a drive failure.) You should also investigate gigabit-tier internet so you can upload these files to an off-site backup if they are important.
You want to go buy a NAS and connect it to your PC. I don’t know know the capacity of a $1,000 NAS and I’m Canadian so I have no idea what stuff costs with real currency.
Backup OFFSITE. Frankly if you could have just 1 huge single drive, and an offsite backup, I’d go with that. A second spare drive in your same system that gets written to can fail many different ways. Offsite backups are much more resilient.
Also Fibre Channel.
I didn’t think 16TB drivers were available yet. Even if they are, the largest sizes are disproportionately more expensive (i.e. it would cost more than twice the cost of two 8TB drives).
RAID0 has no redundancy, so if just one of the drives fails, you lose all the data in the array. Don’t do that unless you don’t care about reliability (e.g. local cache of something that’s stored on a central server).
Is that $1000 for the whole drive/array? At that price point, we’re talking about low-end consumer stuff. The Drobo 5C may be a decent option, it’s a 5-drive RAID-like enclosure with USB 3.0. You can fill it up with five 4TB drives, giving you 16TB storage space for just under $1000 total. It does an equivalent of RAID-5, so it can tolerate failure of 1 drive without any data loss.
The question is what happens WHEN it fails. There is no “if”.
Unless your answer is “That’s OK, we have [solid backup plan]”, you’re not ready.
There is nothing wrong with RAID0 if you have redundancy elsewhere. RAID5 and 6 are very slow for writes due to the striping and parity calculation; that might not be an acceptable tradeoff depending on the application.
That’s true. If I/O matters RAID-6 sucks and RAID-10 is better.
The largest regularly available is 12TB. $468.50 at Newegg for WD Black. If you go that route, definitely use some sort of backup (Backblaze, Carbonite, etc.)
If you can afford 3 drives, probably RAID5. Only 2, RAID1. 4 drives RAID6 provides the best security/laziness tolerance, but for a consumer RAID10 is probably better. They have the same capacity (total storage / 2), but 6 allows 2 failures before you’re borked, and 10 either 1 or 2, depending on which fails.
RAID0 would be for e.g. the HD you run games off of, that already get Steam Cloud backup.
Backup is not needed right now, these files are also located elsewhere and they are almost temporary.
What is the best 12TB drive to get? Do I need a special enclosure to contain it?
I’m not a tech expert at all, but on a recent very long shoot we used a 12 TB external G-raid as the main backup (not sure how configured) and would also dump off X days of footage to a 4-TB shuttle drive that got sent home every few days. We’d then continue to dump to the G-raid and a second shuttle drive until the first shuttle came back from home base, at which point we’d wipe everything on the G-raid that had been ingested back home, and so on. G-raids are pretty standard in my field as a high-quality backup–although one you don’t want to move a lot–and for drives that travel, LaCie Rugged drives are definitely the the thing. 12 TB G-raids go for about $650 US.
I like Western Digital Black. They are their “consumer performance” line. Blue is fine if it ends up being much cheaper. Red if you’re doing NAS or something. Purple is for security cameras. Looks like they don’t make Green anymore? Anyway, WD is one of the better brands, I think Toshiba is good too but no experience. Seagate has become bad lately, at least some models. You might find other brands but they’re rebranded, there are only a few real manufacturers.
You won’t need an enclosure unless you are lacking internal bays or want a travel drive. Some enclosures have a max capacity but motherboards usually don’t.
I think someone linked to this previously. It’s a page containing reliability statistics from Backblaze, an online backup company that has over 100,000 hard drives in use. It might help you decide which to buy.
Get a couple of these babies and your problem is solved. I like the idea of external USB hard drives because you can protect them or safely store them if full by simply unplugging them. Just two of these using just two USB ports will give you 40 TB of storage!
RAID 0 has an extremely significant downside. If any one of the disks comprising the RAID fails, all of the files in the system will be gone.
It’s also true that if you’re not using RAID at all, and have a single hard disk in your computer, then you will lose all your files if the disk fails. Using multiple disks does somewhat increase the chances of a single disk failure. But the OP has already stated that reliability is not a requirement, so it seems like RAID 0 meets his needs for maximum storage.
Raid 0 - striped across multiple disks - yes, if one disk fails, your data is gone.
Raid 5 - striped across 3 or more disks including a checksum on the extra disk - so if one disk fails, the data is not gone. It is in a fragile state, if you lose a second disk before you repace the fist and rebuild the checksums; however, you pay for 3 disks and have the space of 2 (or 4 for 3, 5 for 4, etc.)
Raid 1 is just mirroring - same on each drive.
Other RAID schemes are more complex variations of 5.
There were some options to do RAID and mirroring in software, but generally you need a not-so-cheap disk controller to make it work. (And read the reviews. some cheaper RAID controllers are notoriously unreliable)
IIRC you can also have multiple ordinary drives in your PC and simply extend Windows partitions across multiple drives; again, lose one and the whole is toast.
Absolute simplest is to have 2 drives, and manually copy from primary to backup when you want a “snapshot in time” of your files.
If you are lucky, you can connect two more drives with your internal (SATA we hope) computer disk controller, usually built onto the motherboard.
Some support 2 connections, some support 4; I’m assuming you want to keep this separate from the base Windows hard disk.
If you go external disk, then you will probably want USB3 - if your computer has it, or if not, add a card with that.
Keep in mind 4TB is a heckuva lot of data and will take a LONG time to write or read.
Another option is to have a different set of smaller external disks for each file, if you can use say, 4TB drives and only need to deal with one file at a time.