I’m a photographer and the vast majority of my work is digital. My newest camera produces approximately 20mb RAW files which I convert to DNG and I also create from those JPGs and TIFFs.
So I currently store my collection of files in three places: on the machines I work with, an external hard drive, and on DVD. However I know the information on these won’t last forever. My current plan is to copy these files from the external hard drives to another external hard drive in, say 7 or 8 years time. I’m assuming (or hoping) these new hard drives will be yet bigger and hopefully more reliable so it will become increasingly convenient.
However, what is the BEST but practical way to store my images (or any other data) for, say, another 60 years?
Mods: apologies if this is not considered by you to be a factual question.
There’s no real proven way to store digital data for 60 years. Your best bet is to keep doing what you’re doing, move them from media to media every 7-10 years.
You’ll need to keep refreshing your archive at least once a decade. I doubt any current format will be easily readable in 60 years, even if the media is in perfect condition.
I’m a photographer as well, and face the same dilemma. I make DVDs, backup to external drives, but I fear the only sure way to keep my images viewable is printing them.
One thing I won’t consider is online storage; I see these fear-mongering ads on TV and laugh. Their cost for a year’s storage would buy a nice 250GB drive.
You are doing it the best way in general. I wouldn’t bother with the DVD’s though because they are too small. I would get at least one other external hard drive of a different brand and model and keep rotating backups one of which is stored in a very safe off-site location at any given time. I don’t know what kind of external drive you have now but they have some physically small ones now that don’t require a power cord and they don’t cost very much. Online backup services like www.mozy.com are also good as an additional strategy and you can retrieve your files from anywhere. It is either very cheap or free depending on your needs. Other than that, you can’t just lock electronic media away and forget about it. You have to keep moving it to newer mediums every few years.
Online storage certainly has its place. I recently had my main computer hard drive crash and my external drive also had problems and the backup on it was 6 weeks old. I would have puked right then and there except I had a full backup on Mozy that runs every night and it only takes a few minutes. You can retrieve your files from anywhere with an internet browser. External drives are much faster for large, single backups and for large retrievals but online backups mean that you data will be there even if you have to flee the country suddenly. It only costs about $5 month and that is for their unlimited packages.
There was an interesting article about data storage in a recent issue of American Scientist. It pointed out, among other things, that the density of data storage was increasing , but at the same time the lifetime of the storage media was decreasing very rapidly. None of the current high-density storage methods are rated foir anything like 60 years.
Based on our experience, your best bet might be to transfer your data to cuneiform on baked clay tablets. The storage density is pretty low, and at some point you’ll be forced to hand-transfer data. But you can’t beat the proven storage time of 4000+ years.
When will Google do something akin to a personal data manager on their servers ? You can actually do that with Gmail somehow, but the process of sending yourself your files is horrible and you’re limited to 20MB per send and 7GB total.
Ironically, in switching from film to digital, we gave up a medium that provided archival capabilities of a century or more for one with far less durability and greater expense. And with a negative or transparency, you can just hold it up to the light and see what’s there. With digital media, you have no idea what you have without a lot of extra equipment.
There is an application that turns gmail into a virtual drive and it works well. I save the most critical things to it. I have about 5 levels of backups at this point for everything important. I wouldn’t recommend it a primary or even secondary storage device but it is free so you might as well use it.
Mechanical hard drives are slowly on their way out to be replaced by solid state drives. SSDs have a fairly short lifespan in terms of the number of IO operations before they get flaky but i’m not sure how long the data would last if you were using them for archiving.
The easiest way would be to use online storage, depending on the amount of data you need to keep it will either be free or ‘pretty cheap’. Then your only worry would be the company holding your data going bust so you would probably want to go with someone like Google who should be around for a long while yet.
Whenever this question comes up on Slashdot, the snarky reply is to label it “Scarlet Johannson/Angelina Jolie hardcore hidden camera” then release it onto the peer-to-peer networks. People will share it forever, even once they figure out the title is bogus, and you’ll always be able to find it.
And be sure to pack a spare drive and a computer that it works in.
Never mind 60 years - how do you read 10 year old floppies? Even if you stashed away a internal floppy drive, can you walk into a computer shop today and find anything to connect it to? Might be tough at a “real” computer store, and probably impossible at mass-market places. Similarly, if you put your files on a hard drive and stored it for ten years, got anything now that you can connect it to? The EIDE / PATA interface is getting scarce.
If you’ve got critical data stored on a working computer in the metaphorical deep freeze, and you are good at firing it up every five years or so to refresh the stored data you should be able to transpose it to the next generation of storage. Ethernet will probably still be around in some form so you can transfer the files to the computer of 2100.
Oh, by the way, use the least-complex and least-proprietary format possible. For words, that would be plain text and/or rich text format, and for pictures, try to go with TIFF and uncompressed JPEG.
I’m not an engineer, but does copying date from one hard drive to another actually accomplish anything? If there was corruption in the first hard drive, that corruption is just going to be copied onto the new drive - you’re not fixing any problems that might have arisen.
It also seems like you’re adding in potential fault points. Suppose data gets lost because a drive is defective - the more drives you have in sequence, the greater than chance that you’re going to introduce a defect into the sequence. And there’s also the possibility of corrupting the data during the transfers themselves.
In addition to always getting a copy on new types of media (otherwise, be prepared to save a physical device to read the media, in addition to the media: remember the NASA tapes of the lunar landing: it was difficult to find a device that could read any old tapes they still had?), you should also be prepared to constantly update the files to save them in a current format. There have been many cases where I created a bunch of documents using a software program that didn’t work any more after I changed operating systems, either because the program was no longer being maintained, or because I got a different program to do the same task.
Digital preservation is one of the major challenges facing archives and libraries around the world. The PLANETS project is one of the larger projects but there are several other initiatives like it. AFAIK noone has found the best way to do it but lots of good minds are working on it.
Some European national libraries are using a model involving storing the data on HD in several different physical locations. The drives are connected through some sort of network involving constant (24/7) checksum checks. If and when data in one location suddenly turns up looking different from the rest of the archives, the error is immediately corrected.
Sorry - I don’t know the technical details, but I gather from my colleagues in the preservation dept. that it kinda sorta works as explained above. I do know that it is very, very expensive - thousands of dollars per Tb/year, which is a lot when you are trying to preserve hundres of Tb like many national libraries are. I am not aware of any “home editions” of this equipment.
Until a cheap, safe online archive site appears (and it will), your best bet is to spread your assets: DVD, HD, online - and migrate the data often enough to prevent the media from decaying.
What it accomplishes is geting the data off a drive built in 1998 whose lubrication has just about evaporated. Whose wires (there are a few) have cruddy isulation and some corrrosion. And getting the data onto a drive built in 2010 with nice fresh lube. Lather rinse repeat every few years.
Your point about corruption is well-taken, but that isn’t the real threat. The danger is either old platters which have lost some bits, or old hardware which quits working before you get the data off it. The copying itself can be tested and proven to be perfect. You just have to do it *before *your original becomes damaged in some way.
