I don't want to wait 7 years for a petabyte drive

[Projammer]

I was walking down the hallway yesterday when one of my wororkers asked me what the next step up from a terabyte drive would be.

Me: Umm. That would be a petabyte.
cw: How long before I can buy one?
Me: I'd guess about 7 to 10 years.
cw: Okay, thanks!
Me: Welcome. (strolls away)

I didn't ask why. As far as odd questions around here, that's not even a blip on the radar.

But with the question having been asked, I'm compelled to research and find an answer besides my best guess.

It seems that I was seriously pessimistic, at least according to ZDNet (http://articles.mercola.com/sites/articles/archive/2003/02/22/petabyte.aspx). According to their 2003 article,
At this rate, the one-petabyte (1-million-gigabyte) disk will arrive shortly before 2010. Comparatively, the world disk drive production in 1995 totaled 20 petabytes.

Earlier this year I bought a terabyte drive for about 150. Today they're 100 and 1.5 terabytes can be had for 130.

Based on that, I'm going to have to stick with my original estimate of around 2015.

Now I've been in the business since the 80's so I've seen just about every advancement in storage technology that's made it onto the consumer market.

10meg HD: "Damn, that's nice." But I didn't see it as being unfillable.
20meg: About the same.
120m: Now that's huge. I'll never have to delete anything again.
500m: Okay, that's huge.
1 gig: Lather, rinse, repeat.
100g:
250g:
1t: People stopped worrying about disk management years ago. Run out of space for your pron? Buy a bigger drive.

OS just keeps getting bigger. Apps are bloatware, programmers think code optimization is a waste of time, everythign imaginable gets tacked on because there's plenty of room for it. Games are monstrous with graphics and sound.

I have a PC here that I did a wipe and reload on earlier this week. XP alone tops 5gig of space. It's not unheard of for a game with all the expansion packs topping 100gig.

I know I'm rambling, definately mundane and pointless, but I'm just sitting here drinking my coffee right now deciding where to start my day.

What will we do with a petabyte?

[Sage Rat]

3D video.

[Amp]

Achieve immortality by downloading myself into my PC.

[Ethilrist]

More importantly, how powerful are our machines going to have to be to access that much data in a meaninfully short period of time? One of the advantages of a RAID array giving you a lot of hard drive space is that you have five separate hard drives and, with proper hardware, can access more than one chunk o' data at a time. With a single hard drive holding a petabyte, that's not so much of an option.

If your game is taking up 100g of drive space, and it takes 30 seconds to load, how long is it going to take to load a useful amount off of a drive that holds 10,000 times that much data? Even if we're just thinking of using them for data storage and archiving, that's a lot of time spent transferring data.

[Sage Rat]

If your game is taking up 100g of drive space, and it takes 30 seconds to load, how long is it going to take to load a useful amount off of a drive that holds 10,000 times that much data? Even if we're just thinking of using them for data storage and archiving, that's a lot of time spent transferring data.
Magnetic disk hard drives are likely going to be phased out over the next decade. Solid state technology is already being incorporated as a faster cache within new drives, and will eventually become the full device.

[Kinthalis]

Not only that, but so far higher capacities have meant higher data densities which translate into higher read speeds.

[Sage Rat]

Not only that, but so far higher capacities have meant higher data densities which translate into higher read speeds.
Mmm, yeah sorta. But it hasn't scaled with use so mostly what it has meant is that you get a delay on startup as key data is cached into RAM.

[Smeghead]

As long as I can keep transferring all my downloaded TV shows from stacks and stacks of CDs to one small little box, I'm happy.

[Quartz]

We're actually approaching a bit of a limit because of error rates. An uncorrectable error rate of 1 in 1015 (for the Maxtor Diamondmax I have on the table here) sounds very low, but a 1 terabyte drive has approx 1013 bits, so there's a 1% chance of an uncorrectable error at the disk level. And at that error rate, a 1 petabyte drive will have 10 errors on it. So file systems will have to evolve to deal with this issue or drives have to get more reliable.

[gotpasswords]

3D video.
Close. 3D porn. :D

[neutron star]

What games take up 100GB? That's like 20 DVDs!

[Rigamarole]

I've always wondered why computer upgrades (not just hard drive space but processor speed, memory, etc.) come in such terribly predictable sequence. Back in the 90s when a 256Mhz processor with 12MB of RAM and 120MB hard drive space was top-of-the-line, why couldn't they have started work on developing a computer to today's specs then? Why can't the next computer they develop be a 50Ghz processor with 100GB of RAM and 8 petabytes of hard drive space?

I suspect it has to do with making sure people have to buy a new computer every 2-3 years regardless of true advances in software. Some of the new games coming out (Far Cry 2, The Witcher) run a little choppy on my current CPU (2.8Ghz 1 GB RAM, GeForce 6800) even at their very lowest graphics settings, which don't make them look much better than games of 5-6+ years ago.

[hajario]

Magnetic disk hard drives are likely going to be phased out over the next decade. Solid state technology is already being incorporated as a faster cache within new drives, and will eventually become the full device.

You do realize that they've been saying that for over 20 years now. I was an engineer in the disk drive industry for most of the 90's. In 1990 Maxtor came out with the first 100 Mbyte drive for under $100. Less than a dollar a Mb was nearly unbelievable. Now it's a tiny fraction of a cent.

[Diz]

I've always wondered why computer upgrades (not just hard drive space but processor speed, memory, etc.) come in such terribly predictable sequence. Back in the 90s when a 256Mhz processor with 12MB of RAM and 120MB hard drive space was top-of-the-line, why couldn't they have started work on developing a computer to today's specs then? Why can't the next computer they develop be a 50Ghz processor with 100GB of RAM and 8 petabytes of hard drive space?

That's like asking why Henry Ford didn't start cranking out Mustangs as soon as the Model T ended production. There are a lot of intermediate steps to take before the technology is mature enough. It takes years to solve the original problems, and years more to solve the next set, etc...

Or am I being wooshed? :)

[Rigamarole]

That's like asking why Henry Ford didn't start cranking out Mustangs as soon as the Model T ended production. There are a lot of intermediate steps to take before the technology is mature enough. It takes years to solve the original problems, and years more to solve the next set, etc...

Or am I being wooshed? :)

No I'm genuinely curious because I don't really understand what the process entails. I can more easily and visually imagine the process of designing and producing a Mustang, while the only difference I see when I go to buy a computer today than I would have seen 15 years ago is a few key numbers are higher. I'm sure there's tons and tons of work involved, I just don't know what it is.

[Projammer]

What games take up 100GB? That's like 20 DVDs!

Not quite 100 gig out of the box, but

Age of Conan (http://www.pcgamerequirements.com/category/a-e/) lists 30 gig as a system requirement.

X-Plane (http://www.x-plane.com/sysreq.html) wants 70.

I was actually refering to base game installations plus addons. Maps for Battlefield 2 and the like. Or expansion packs that require the original game as well. Warhammer 40k is an example. You have to have something like 4 or 5 packages installed to have access to all races.

[Projammer]

Too late for an ETA, but.

Can't find stats, but I shudder to think what a full Sims 2 installation might look like.

I'll see what the SO's WoW directory looks like when I get home

[Diz]

No I'm genuinely curious because I don't really understand what the process entails. I can more easily and visually imagine the process of designing and producing a Mustang, while the only difference I see when I go to buy a computer today than I would have seen 15 years ago is a few key numbers are higher. I'm sure there's tons and tons of work involved, I just don't know what it is.

Well the very basic issues involve making tiny things even smaller and moving increasingly large amounts of data even faster. All sorts of problems need to be worked out: heat dissipation, energy use, the actual process of manufacturing something so intricate without it costing a fortune. The finished product may not look all that different, but the technological hurdles overcome are significant. I'm sure someone far more qualified than I could explain the specifics.

Just for comparison, a Pentium 2 CPU from the late '90's had 7.5 million transistors in it. Intel's newly released Core i7 has 731 million.

[Sage Rat]

You do realize that they've been saying that for over 20 years now. I was an engineer in the disk drive industry for most of the 90's. In 1990 Maxtor came out with the first 100 Mbyte drive for under $100. Less than a dollar a Mb was nearly unbelievable. Now it's a tiny fraction of a cent.
Yeah but necessity is the mother of invention. You have to meet a certain minimum of response time. Users aren't going to accept a five minute load time for each game level, or a ten minute load time for Adobe Acrobat. And caching information is the only solution to the problem of the relative slowness of the hard drive, so once that does start to take so long that it becomes unreasonable, things will switch over regardless of anything.

New technology pretty much always sits at the sidelines until the old stuff becomes infeasible. But as soon as the old stuff does become so, the new technology jumps in pretty immediately. There's no reason to think that hard drives won't be the same.

[Eonwe]

What does a petabyte? Vegetables!!! :p

[Jragon]

Too late for an ETA, but.

Can't find stats, but I shudder to think what a full Sims 2 installation might look like.

I'll see what the SO's WoW directory looks like when I get home

Sims 2, all expansions and one fun pack - no addons 10.8GB. The fun packs aren't too big, I can't imagine the number getting more than 12-15.

WoW, with addons (typical users have addons so I'll include it) - 11.7GB

[neutron star]

You do realize that they've been saying that for over 20 years now. I was an engineer in the disk drive industry for most of the 90's. In 1990 Maxtor came out with the first 100 Mbyte drive for under $100. Less than a dollar a Mb was nearly unbelievable. Now it's a tiny fraction of a cent.True, but when they were saying those things back then, they didn't have 128GB solid state drives that use the same interface as regular hard drives (http://www.allstarshop.com/shop/product.asp?pid=20923&ad=pwatch) already on the market.

[Sage Rat]

True, but when they were saying those things back then, they didn't have 128GB solid state drives that use the same interface as regular hard drives (http://www.allstarshop.com/shop/product.asp?pid=20923&ad=pwatch) already on the market.
Ooh, I hadn't seen that one. $319 for 128GB is starting to get pretty reasonable. I looked for something like a year ago and the prices were probably double that.

[clayton_e]

And hajario, its already happening with ipods. I bought an 80 gig a year or so ago for.. what? 250 bucks? Now its non-existant and the 120 gig replaced it.

After that, no more magnetic hard drive ipods. Mine will be a dinosaur in a couple years.. Statistically prone to breakage due to moving parts compared to the solid state pods.

This is happening with laptops for the same reason, being portable they're more prone to breakage. A few, I believe, are already on the market (http://www.buy.com/prod/samsung-x360-34p-laptop-intel-core-2-duo-u9400-1-4ghz-3gb-128gb-solid/q/loc/101/209849660.html) and have been since early 2006 (http://gizmodo.com/gadgets/laptops/samsung-q30-solid-state-laptop-175714.php), at least.

When the cost of production comes down the reliability will make it a nice option for desktops as well. No moving parts means no breaking from physical wear out.

We'll be saying goodbye to our magnetic drives and their laughable needles the same way we did to vinyl records.

[neutron star]

Ooh, I hadn't seen that one. $319 for 128GB is starting to get pretty reasonable. I looked for something like a year ago and the prices were probably double that.
Yup, and a 64GB SSD is only $135. Next time I build myself a computer (should be soon, as I'm still running a freakin' Athlon 1800), I think I'm going to get one to use as my system drive. I wonder how fast one of those things boots Windows.

[Projammer]

Sims 2, all expansions and one fun pack - no addons 10.8GB. The fun packs aren't too big, I can't imagine the number getting more than 12-15.

WoW, with addons (typical users have addons so I'll include it) - 11.7GB

Actually surprised about Sims 2. I go to the store and see close to a dozen different expansions and just assume they're like expansions for other games where it's pretty much all new contents and objects just running on the same engine. Color me educated.

SO's WoW is somewhere around 16.5g. That will be with all the expansions including Lich King because I bought it for her. I know she runs addons, just not which ones.

[Dangerosa]

More importantly, how powerful are our machines going to have to be to access that much data in a meaninfully short period of time? One of the advantages of a RAID array giving you a lot of hard drive space is that you have five separate hard drives and, with proper hardware, can access more than one chunk o' data at a time. With a single hard drive holding a petabyte, that's not so much of an option.

If your game is taking up 100g of drive space, and it takes 30 seconds to load, how long is it going to take to load a useful amount off of a drive that holds 10,000 times that much data? Even if we're just thinking of using them for data storage and archiving, that's a lot of time spent transferring data.

Hybrid flash. Flash doesn't scale as well as magnetic (price per gb) at this point in time - but its really useful for accessing data quickly - so you "cache" onto the flash portion of the drive the parts of the game that require quick access - the rest sits on the hard drive. You combine fast with cheap.

I think you can buy hybrid flash drives now.

[hajario]

True, but when they were saying those things back then, they didn't have 128GB solid state drives that use the same interface as regular hard drives (http://www.allstarshop.com/shop/product.asp?pid=20923&ad=pwatch) already on the market.

They didn't even have 128 GB magnetic HDDs then either.

We'll be saying goodbye to our magnetic drives and their laughable needles the same way we did to vinyl records.

Maybe. I went to my first talk on the end of spinning magnetic disk drives when I was a grad student in a HDD industry supported research lab in 1988. I agree that today's solid state is a way better candidate than bubble memory or optical drives of the past. I also acknowledge that I've been out of that industry for ten years so I'm not at all up on things.

[dwc1970]

If 1.5 TB is the current maximum and if Moore's Law (http://en.wikipedia.org/wiki/Moore%27s_Law) still holds we should see petabyte drives come onto the market around 2023 or so.

[beowulff]

If 1.5 TB is the current maximum and if Moore's Law (http://en.wikipedia.org/wiki/Moore%27s_Law) still holds we should see petabyte drives come onto the market around 2023 or so.

Moore's law describes microprocessor density, not magnetic storage. Hard drives increase in density much faster than semiconductors, which is why I don't see solid state storage displacing it for a long. long time.

[neutron star]

They didn't even have 128 GB magnetic HDDs then either.
My point was that they didn't have solid state drives on the market that were about 1/10th the size of the largest magnetic drives.

[eldowan]

But we will also have to consider the wrap that solid state has for limited writes.
Wasn't the upper limit for the amount of times a specific piece of the flash memory can be overwritten before it degrades to be nearly unusable something in the low 100k ?

I don't believe magnetic storage devices have this kind of limitation, but someone please correct me if I am misinformed.

[hajario]

Moore's law describes microprocessor density, not magnetic storage.

It does a good job describing magnetic storage density too. I believe that Paul Frank first pointed that out.

My point was that they didn't have solid state drives on the market that were about 1/10th the size of the largest magnetic drives.

I see. That does make sense.

[beowulff]

Never mind.

[Sage Rat]

Wasn't the upper limit for the amount of times a specific piece of the flash memory can be overwritten before it degrades to be nearly unusable something in the low 100k ?
If you look at the product linked to in post #22, you'll see that it includes the total number of rewrites (1,000,000), which seems reasonable.