# How can Wikipedia save all that data? **URL:** \ **Category:** Factual Questions\ **Created:** [April 28, 2021, 3:35am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899 "2021-04-28T03:35:49Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Northern\_Piper](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/northern_piper/32/5304_2.png) [@Northern\_Piper](https://boards.straightdope.com/u/Northern_Piper)\ **Post date:** [April 28, 2021, 3:35am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/1 "2021-04-28T03:35:50Z")
Do they have server farms all over the world? How do they do it? ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [April 28, 2021, 3:42am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/2 "2021-04-28T03:42:28Z")
Yes they do have a bunch of servers all over the world: > **[Wikimedia servers - Meta](https://meta.wikimedia.org/wiki/Wikimedia_servers)** Cluster locations: [https://wikitech.wikimedia.org/wiki/Clusters](https://wikitech.wikimedia.org/wiki/Clusters) ---
**Author:** ![Darren\_Garrison](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/darren_garrison/32/92_2.png) [@Darren\_Garrison](https://boards.straightdope.com/u/Darren_Garrison)\ **Post date:** [April 28, 2021, 4:13am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/3 "2021-04-28T04:13:42Z")
> As of 20 April 2021, the size of the current version of all articles compressed is about 19.52 GB (From [here](https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia).) That isn’t including all the photos and video and audio clips. I don’t how much they add, but it is probably pretty safe to say that with even all of that it is a relatively trivial amount of data compared to the truly large sites (Google, Facebook, Netflix, etc.) I’d be suprised if it didn’t _all_ fit on a single modern hard drive. ---
**Author:** ![Darren\_Garrison](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/darren_garrison/32/92_2.png) [@Darren\_Garrison](https://boards.straightdope.com/u/Darren_Garrison)\ **Post date:** [April 28, 2021, 4:21am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/4 "2021-04-28T04:21:55Z")
Apparently if you want the entire history of every edit, it does add up to “terabytes” uncompressed, but it doesn’t specify how many: > **[Wikipedia:Database download](https://en.wikipedia.org/wiki/Wikipedia:Database_download)** > > Pages for logged out editors learn more > > Wikipedia offers free copies of all available content to interested users. These databases can be used for mirroring, personal use, informal backups, offline use or database queries (such as for Wikipedia:Maintenance). All text content is licensed under the Creative Commons Attribution-ShareAlike 3.0 License (CC-BY-SA), and most is additionally licensed under the GNU Free Documentation License (GFDL). Images and other files are available under dif... (The off-line version of text/tables only Wikipedia that I downloaded multiple years ago was only 10 GB compressed, 20 GB expanded and indexed. I read it with [this](https://wikitaxi.en.softonic.com/).) ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [April 28, 2021, 4:30am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/5 "2021-04-28T04:30:28Z")
It is definitely not “big data”. And an off-line version, text only, is really nothing; you could save it to your smartphone. Even if you want all the languages, all the articles, plus user/talk pages, plus the entire history of all of it, including all the images, videos, and other media, you could keep a copy at home, though you might consider budgeting 100 TB to be on the safe side. ---
**Author:** ![Darren\_Garrison](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/darren_garrison/32/92_2.png) [@Darren\_Garrison](https://boards.straightdope.com/u/Darren_Garrison)\ **Post date:** [April 28, 2021, 4:40am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/6 "2021-04-28T04:40:48Z")
> [@DPRK](#): > > And an off-line version, text only, is really nothing; you could save it to your smartphone. I wanted one of these when they were around: > **[Amazon.com : WikiReader Pocket Wikipedia : Electronic Reference Devices : Office...](https://www.amazon.com/WikiReader-PANREADER-Pocket-Wikipedia/dp/B002N5521W)** > > WikiReader delivers the joy of information by offering three million Wikipedia articles in one simple $99 handheld device. It turns on instantly and runs for months before its two AAA batteries need to be replaced. The large monochrome screen uses a... ---
**Author:** ![BigT](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/bigt/32/12044_2.png) [@BigT](https://boards.straightdope.com/u/BigT)\ **Post date:** [April 28, 2021, 5:12am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/7 "2021-04-28T05:12:52Z")
> [@Darren\_Garrison](#): > > > [@DPRK](#): > > > > And an off-line version, text only, is really nothing; you could save it to your smartphone. > > I wanted one of these when they were around: > > [https://www.amazon.com/dp/B002N5521W/](https://www.amazon.com/dp/B002N5521W/) I actually have a Kindle 2 that Randal Monroe went on about being a permanent Wikipedia device, with free data. Unfortunately, the idea that every single site needed to be encrypted and the fact that encryption protocols change over time ruined that part of it. But, last I checked, it still does get free data over cellular. > **[Kindle](https://xkcd.com/548/)** > > I'm happy with my Kindle 2 so far, but if they cut off the free Wikipedia browsing, I plan to show up drunk on Jeff Bezos's lawn and refuse to leave. ---
**Author:** ![psychonaut](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/psychonaut/32/4655_2.png) [@psychonaut](https://boards.straightdope.com/u/psychonaut)\ **Post date:** [April 28, 2021, 8:37am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/8 "2021-04-28T08:37:34Z")
> [@DPRK](#): > > It is definitely not “big data”. Oh, it definitely is “big data”. Hundreds, maybe even thousands, of scientific articles have been written that explicitly use this term to describe data from Wikipedia. ---
**Author:** ![Schnitte](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/schnitte/32/9033_2.png) [@Schnitte](https://boards.straightdope.com/u/Schnitte)\ **Post date:** [April 28, 2021, 1:03pm UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/9 "2021-04-28T13:03:55Z")
You can surely draw big data from Wikipedia, as, for instance, [this article](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0071226) did (making predictions of the success of an upcoming film based on user activity on Wikipedia). But that doesn’t mean Wikipedia’s own content is big data itself. ---
**Author:** ![Kimera757](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/kimera757/32/3765_2.png) [@Kimera757](https://boards.straightdope.com/u/Kimera757)\ **Post date:** [April 28, 2021, 1:16pm UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/10 "2021-04-28T13:16:15Z")
I assume there’s a similar answer for [Archive.org](http://Archive.org). It’s based in San Francisco but simply says it has many servers. The old set were replaced in 2010, which causes me to wonder just how much maintenance a massive server farm requires. Same thing for Google, Facebook, Amazon data servers, etc. ---
**Author:** ![bump](https://avatars.discourse-cdn.com/v4/letter/b/7c8e57/32.png) [@bump](https://boards.straightdope.com/u/bump)\ **Post date:** [April 28, 2021, 2:45pm UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/11 "2021-04-28T14:45:08Z")
Exactly. ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [April 28, 2021, 5:40pm UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/12 "2021-04-28T17:40:30Z")
Slightly large data? There is a lot of it, in human terms. You may need to employ non-trivial algorithms to extract information from it. But the data set fits into your pocket. ---
**Author:** ![DPRK](https://avatars.discourse-cdn.com/v4/letter/d/4491bb/32.png) [@DPRK](https://boards.straightdope.com/u/DPRK)\ **Post date:** [April 28, 2021, 5:54pm UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/13 "2021-04-28T17:54:36Z")
> [@Kimera757](#): > > I assume there’s a similar answer for [Archive.org](http://Archive.org). > > It’s based in San Francisco but simply says it has many servers. The old set were replaced in 2010, which causes me to wonder just how much maintenance a massive server farm requires. > > Same thing for Google, Facebook, Amazon data servers, etc. Wikipedia, though, is not big enough to require maintaining a server farm. They pay to rent facilities, or receive such services as a donation. The ones maintaining the massive server farms are Equinix, EvoSwitch, CyrusOne, and UnitedLayer. The Internet Archive is bigger data (~100 PB) and I believe hosted it themselves at least at one point. They somehow manage as a non-profit with modest revenue. ---
**Author:** ![Darren\_Garrison](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/darren_garrison/32/92_2.png) [@Darren\_Garrison](https://boards.straightdope.com/u/Darren_Garrison)\ **Post date:** [April 28, 2021, 5:55pm UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/14 "2021-04-28T17:55:14Z")
> [@DPRK](#): > > Slightly large data? There is a lot of it, in human terms. You may need to employ non-trivial algorithms to extract information from it. But the data set fits into your pocket. The concept of “big data” must necessarily change over time, just like the concept of a “supercomputer”. Back when Wikipedia started, a dataset of 2 or 3 or 5 TB definitely counted as big data. Now it is a couple hundred bucks worth of storage that you can (probably) buy at your local Wal-Mart. ---
**Author:** ![suranyi](https://avatars.discourse-cdn.com/v4/letter/s/e36b37/32.png) [@suranyi](https://boards.straightdope.com/u/suranyi)\ **Post date:** [April 28, 2021, 6:47pm UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/15 "2021-04-28T18:47:04Z")
Reminds me of when I started working on electronic road maps for car navigation systems, back in the mid 90s. We had to split California into two CD-ROMs (remember them?) because it was too big to fit on one. Still, I was amazed that all of Northern Cal — every single street — could fit on one disk. Nowadays, of course, Google and Apple hold pretty much every street in the world, and far more besides, in their map databases. And it’s not considered truly big data compared to what else they have. ---
**Author:** ![Darren\_Garrison](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/darren_garrison/32/92_2.png) [@Darren\_Garrison](https://boards.straightdope.com/u/Darren_Garrison)\ **Post date:** [April 28, 2021, 9:55pm UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/16 "2021-04-28T21:55:53Z")
I used to have Street Atlas USA on one CD. Whole country. Roads only, no gas stations, etc. ---
**Author:** ![LSLGuy](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/lslguy/32/5813_2.png) [@LSLGuy](https://boards.straightdope.com/u/LSLGuy)\ **Post date:** [April 28, 2021, 10:50pm UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/17 "2021-04-28T22:50:06Z")
Here’s some xkcd wisdom about the size of wiki: [Updating a Printed Wikipedia](https://what-if.xkcd.com/59/) As I see it, the real point of the thought experiment in my cited article is to help us bridge the monumental gap between what constitutes “a lot of data” in some physical human readable human touchable format versus “a lot of data” in modern standard computer-accessible formats. Storing a printed complete wikipedia would require a large warehouse. Storing a computerized complete wikipedia requires a large pocket. That’s hard to wrap our minds around. ---
**Author:** ![Northern\_Piper](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/northern_piper/32/5304_2.png) [@Northern\_Piper](https://boards.straightdope.com/u/Northern_Piper)\ **Post date:** [April 28, 2021, 10:53pm UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/18 "2021-04-28T22:53:37Z")
Fascinating. Thank you all. I had no idea how compact the storage system has become. I will go back to running my programs on my punch cards now. ---
**Author:** ![DMC](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dmc/32/18049_2.png) [@DMC](https://boards.straightdope.com/u/DMC)\ **Post date:** [April 29, 2021, 1:15am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/19 "2021-04-29T01:15:42Z")
> [@psychonaut](#): > > Hundreds, maybe even thousands, of scientific articles have been written that explicitly use this term to describe data from Wikipedia. And hundreds, maybe even thousands, of scientific articles (and marketers, more importantly) can’t even agree on what is meant by “big data”. [![](https://image.slidesharecdn.com/symfonycon-bigdata-141128022132-conversion-gate01/95/big-data-great-now-what-symfonycon-2014-5-638.jpg?cb=1417176044) ](https://image.slidesharecdn.com/symfonycon-bigdata-141128022132-conversion-gate01/95/big-data-great-now-what-symfonycon-2014-5-638.jpg?cb=1417176044) ---
**Author:** ![DMC](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dmc/32/18049_2.png) [@DMC](https://boards.straightdope.com/u/DMC)\ **Post date:** [April 29, 2021, 1:40am UTC](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899/20 "2021-04-29T01:40:12Z")
Back to the OP. As @DPRK and @Darren_Garrison noted, the volume of data isn’t all that big. The primary reason they need hundreds of servers is because of the number of people looking at that data. A few years ago they were already serving over 600 million page views per day. I think the primary backend db is still MariaDB, which is just a fork of MySQL, and it’s going to have trouble keeping up with tens of thousands of simultaneous query hits on a single server. [Next page](https://boards.straightdope.com/t/how-can-wikipedia-save-all-that-data/940899.md?page=2)