# How can Wikipedia save all that data? **URL:** \ **Category:** Factual Questions\ **Created:** [April 28, 2021, 3:35am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899 "2021-04-28T03:35:49Z")\ **Posts on this page:** 8\ **Page:** 2
**Author:** ![Darren\_Garrison](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/darren_garrison/32/92_2.png) [@Darren\_Garrison](https://boards.straightdope.com/u/Darren_Garrison)\ **Post date:** [April 29, 2021, 1:41am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/21 "2021-04-29T01:41:02Z")
> [@LSLGuy](#): > > Storing a printed complete wikipedia would require a large warehouse. Storing a computerized complete wikipedia requires a large pocket. That’s hard to wrap our minds around. Putting it a different way, checking in on a torrent site, I see that season one of _Game of Thrones_ in 4K video with Dolby Atmos sound is 331 GB. A single season of TV ten times the size of the current text of Wikipedia. ---
**Author:** ![Darren\_Garrison](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/darren_garrison/32/92_2.png) [@Darren\_Garrison](https://boards.straightdope.com/u/Darren_Garrison)\ **Post date:** [April 29, 2021, 1:49am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/22 "2021-04-29T01:49:16Z")
> [@Northern\_Piper](#): > > Fascinating. Thank you all. I had no idea how compact the storage system has become. See: > **[Western Digital releases new 18TB, 20TB EAMR drives](https://arstechnica.com/gadgets/2020/07/western-digital-releases-new-18tb-20tb-eamr-drives/)** > > EAMR offers similar density gains to HAMR and MAMR, with less exotic tech. > **[At 100TB, the world’s biggest SSD gets an (eye-watering) price tag](https://www.techradar.com/news/at-100tb-the-worlds-biggest-ssd-gets-an-eye-watering-price-tag)** > > Largest ever solid state drive comes with very steep premium ---
**Author:** ![LSLGuy](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lslguy/32/5813_2.png) [@LSLGuy](https://boards.straightdope.com/u/LSLGuy)\ **Post date:** [April 29, 2021, 2:24am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/23 "2021-04-29T02:24:25Z")
Ref that second item. It’s about 4" by 6" x 1", weighs a few ounces, and stores 100TB for sub-millisecond access. And costs what modern people think is a lot of money for a gizmo at US$12,500. I can recall speccing 100MB drives being $50K each and we’d need a million of them to hold the same data. Each one the size of a typical US home washing machine. A million washing machines stacked tightly is a rectangular block about 250 x 300 x 400 _feet_ tall. Plus the home refrigerator-sized controller box needed for every 6 washing machines. So toss 180,000 of those on the pile too. And the even larger “channel devices” to control 15 controllers each, so 12,000 of those. And lots (!!1!) of cables as wide as your forearm. It’ll cost a mere 50 _billion_ dollars for the drives before you buy the even more eye-wateringly expensive controllers and channels, or before you build the VAB-sized structure\* to hold it, or pay an army of men (yes, all men) in crew cuts, white shirts, and narrow black ties to wire it all together. Gawd help the guy who has to pay the power bill. The thing would literally melt into a heap of slag from the concentrated heat unless we spread the boxes out probably 2x or 3x as large in each dimension overall and provided insane amounts of HVAC capacity. Hell, we’d probably need cyrogenics to prevent a meltdown. Or, for $12 grand (plus tax), stop by the store, pick one up, and stick it in your purse on the way out the door. With room for a bag of the cheap impulse buy candy they keep by the cash register. Easy peasy. We live in miraculous times; truly we do. * * * \* VAB: > **[Vehicle Assembly Building](https://en.wikipedia.org/wiki/Vehicle_Assembly_Building)** > > Pages for logged out editors learn more > > The Vehicle Assembly Building (originally the Vertical Assembly Building), or VAB, is a large building at NASA's Kennedy Space Center (KSC), designed to assemble large pre-manufactured space vehicle components, such as the massive Saturn V and the Space Shuttle, and stack them vertically onto one of three mobile launcher platforms used by NASA. As of March 2022, the first Space Launch System (SLS) rocket was assembled inside in preparation for th... ---
**Author:** ![Darren\_Garrison](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/darren_garrison/32/92_2.png) [@Darren\_Garrison](https://boards.straightdope.com/u/Darren_Garrison)\ **Post date:** [April 29, 2021, 2:38am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/24 "2021-04-29T02:38:46Z")
> [@LSLGuy](#): > > We live in miraculous times, truly we do. I often think about the irony of how useful something like the Cray 1 was for some scientific fields yet simultaneously how completely useless that amount of MIPS/RAM/storage is for most mundane real-world home computer uses. Sure, you can do weather prediction and nuclear weapons design, but can you imagine the patience required to attempt to edit a 50 MP JPEG on one of [these](https://en.wikipedia.org/wiki/Cray-1) turkeys? ---
**Author:** ![Dewey\_Finn](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dewey_finn/32/4222_2.png) [@Dewey\_Finn](https://boards.straightdope.com/u/Dewey_Finn)\ **Post date:** [April 29, 2021, 2:41am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/25 "2021-04-29T02:41:20Z")
> [@LSLGuy](#): > > Ref that second item. It’s about 4" by 6" x 1", weighs a few ounces, and stores 100TB for sub-millisecond access. And costs what modern people think is a lot of money for a gizmo at US$12,500. I’m trying to imagine how much storage you would get if you filled a rack with those 100TB drives. ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [April 29, 2021, 2:55am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/26 "2021-04-29T02:55:59Z")
> [@Dewey\_Finn](#): > > I’m trying to imagine how much storage you would get if you filled a rack with those 100TB drives. That is easy to calculate, though there are non-trivial considerations involved. An Internet Archive-style “PetaBox” has 240 disks per rack. If each drive contains 100 TB, that makes 24 PB per rack. The maximum amount you can actually store depends on the RAID configuration, number of disks that are spares, type of filesystem, etc. > [@LSLGuy](#): > > We live in miraculous times; truly we do. The old-timers are always talking about how a 32-bit memory module cost $400 in the 1950s, or whatever. Although increasing speed and capacity can unequivocally make once-big data no longer big, one thing I have heard is that it is not exactly the physical size of the storage unit that makes a set “big”, but the types and sophistication of the algorithms required to extract useful information from it; even sorting and searching start to run into bottlenecks if there are enough elements. ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [April 29, 2021, 3:07am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/27 "2021-04-29T03:07:12Z")
It’s all about algorithmic complexity. Better supercomputer ==\> more lattice points in your physical simulation, or more pixels in your JPEG… You can crank up some parameter _n_ . ---
**Author:** ![LSLGuy](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lslguy/32/5813_2.png) [@LSLGuy](https://boards.straightdope.com/u/LSLGuy)\ **Post date:** [April 29, 2021, 3:21am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/28 "2021-04-29T03:21:40Z")
Yeah. We’ve also made great strides in algorithms from the basics of the 1940s & 50s. But … My never-very-expert-and-now-musty intuition is that we’ve made more progress in the capacity to gather data at scale and to store data at scale than we have in the ability to get useful results at scale. We may soon be in a literal situation where our reach exceeds our grasp. We can observe everything and record everything, but we can’t make useful sense of most of it. OTOH, quantum physics suggests we’ve wrung almost all the juice out of Moore’s law that there is to be had. He’s running dry. So something other than the hardware price/performance gains I outlined upthread will be needed to keep powering the IT miracle towards the future. If it isn’t quantum magic it’s got to be algorithm design. [Previous page](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899.md?page=1)