# Deep Web Search Engines **URL:** \ **Category:** Factual Questions\ **Created:** [June 20, 2004, 11:18am UTC](https://boards.straightdope.com/t/deep-web-search-engines/251295 "2004-06-20T11:18:08Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Go\_You\_Big\_Red\_Fire\_Engine](https://avatars.discourse-cdn.com/v4/letter/g/eada6e/32.png) [@Go\_You\_Big\_Red\_Fire\_Engine](https://boards.straightdope.com/u/Go_You_Big_Red_Fire_Engine)\ **Post date:** [June 20, 2004, 11:18am UTC](https://boards.straightdope.com/t/deep-web-search-engines/251295/1 "2004-06-20T11:18:08Z")
I tried googlin’ for one, and unsurprisingly enough, came up with nothing. I have heard about the “wonders” of the deep web, and want to try it out. Any recommendations, or is it not worth the trouble? I considered putting this in IMHO, but I do want factual answers and statistics. Our engineering lecturer said something along the lies of “google only contains 3% of the available content on the web.” Verification? ---
**Author:** ![alterego](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@alterego](https://boards.straightdope.com/u/alterego)\ **Post date:** [June 20, 2004, 11:44am UTC](https://boards.straightdope.com/t/deep-web-search-engines/251295/2 "2004-06-20T11:44:08Z")
I have a link to a deep-web website in the [references sticky](http://boards.straightdope.com/sdmb/showthread.php?t=149223&page=3). There was a book written on it and they published a website alongside it. It really hasn’t been updated since 2001 so some of the search links might not be accurate, however they accounted for this by simply providing links to the websites as well the website’s search engines. IMHO, I don’t recommend the book. I also want to point out that by its nature the deep web has no search engines. See this quote from [Wikipedia](http://en.wikipedia.org/wiki/Deep_web): > [@](#): > > The deep web or invisible web is the name given to the publicly accessible pages on the World Wide Web that are not indexed by search engines. It consists mainly of dynamically generated pages that are based on responses to database queries. While you can be pointed towards websites that are members of the deep web, once they have been spidered by a search engine (search engines run by the very site notwithstanding) they are no longer part of the deep web! In reference to your sig, “Choke a fish or kill a tree?” 🆒 ---
**Author:** ![alterego](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@alterego](https://boards.straightdope.com/u/alterego)\ **Post date:** [June 20, 2004, 11:45am UTC](https://boards.straightdope.com/t/deep-web-search-engines/251295/3 "2004-06-20T11:45:51Z")
Sorry - the website is [http://www.invisible-web.net/](http://www.invisible-web.net/) ---
**Author:** ![Go\_You\_Big\_Red\_Fire\_Engine](https://avatars.discourse-cdn.com/v4/letter/g/eada6e/32.png) [@Go\_You\_Big\_Red\_Fire\_Engine](https://boards.straightdope.com/u/Go_You_Big_Red_Fire_Engine)\ **Post date:** [June 20, 2004, 11:48am UTC](https://boards.straightdope.com/t/deep-web-search-engines/251295/4 "2004-06-20T11:48:25Z")
That’s what googling turned up, but it didn’t help me much. But your post does make so much sense. I feel really dumb now. :smack: In response to my sig, I don’t use paper bags either? It’s all backpack stuffing for me 🙂 ---
**Author:** ![Go\_You\_Big\_Red\_Fire\_Engine](https://avatars.discourse-cdn.com/v4/letter/g/eada6e/32.png) [@Go\_You\_Big\_Red\_Fire\_Engine](https://boards.straightdope.com/u/Go_You_Big_Red_Fire_Engine)\ **Post date:** [June 20, 2004, 11:50am UTC](https://boards.straightdope.com/t/deep-web-search-engines/251295/5 "2004-06-20T11:50:21Z")
I take that back. I was at [http://invisibleweb.com](http://invisibleweb.com) :smack: ---
**Author:** ![alterego](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@alterego](https://boards.straightdope.com/u/alterego)\ **Post date:** [June 20, 2004, 11:57am UTC](https://boards.straightdope.com/t/deep-web-search-engines/251295/6 "2004-06-20T11:57:00Z")
That sucks. Damn those squatters! ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [June 20, 2004, 7:08pm UTC](https://boards.straightdope.com/t/deep-web-search-engines/251295/7 "2004-06-20T19:08:31Z")
There’s also the matter that most of the deep internet is specialized information. For instance, a researcher might have his computer hooked up to the Internet so he can run his simulations remotely, using a password. That computer is then part of the deep internet. But only people who have a reason to use it can access it. ---
**Author:** ![alterego](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@alterego](https://boards.straightdope.com/u/alterego)\ **Post date:** [June 20, 2004, 8:01pm UTC](https://boards.straightdope.com/t/deep-web-search-engines/251295/8 "2004-06-20T20:01:21Z")
> [@Chronos](#): > > There’s also the matter that most of the deep internet is specialized information. For instance, a researcher might have his computer hooked up to the Internet so he can run his simulations remotely, using a password. That computer is then part of the deep internet. But only people who have a reason to use it can access it. From everything I have read, including the book above, it only applies to publicly accessible web sites that a search engine could not possibly crawl because the pages are generated dynamicly and simply do not exist in static form - thereby making themselves invisible or “deep”. To requote Wikipedia: > [@](#): > > The deep web or invisible web is the name given to the **publicly accessible** pages on the World Wide Web that are not indexed by search engines. It consists mainly of dynamically generated pages that are based on responses to database queries. A definition given by [Berkeley](http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html) seems to imply the same - pages that are searchable by the end user: > [@](#): > > The “invisible web” is what you cannot retrieve (“see”) in the search results and other links contained in these types of tools. > > ``` > * **Searchable Databases**. Most of the invisible web is made up of the contents of thousands of specialized searchable databases **that you can search via the Web**. The search results from many of these databases are delivered to you in web pages that are generated just in answer to your search. Such pages very often are not stored anywhere: it is easier and cheaper to dynamically generate the answer page for each query than to store all the possible pages containing all the possible answers to all the possible queries people could make to the database. Search engines cannot find or create these pages. More explanation. > > * **Excluded Pages**. There are some types of pages that search engine companies exclude by policy. There is no technical reason they could not include them if they wanted. It's a matter of selecting what and what not to include in databases that are already huge, expensive to operate, and whose search function is a low revenue producer. More explanation. > > ``` Although it could be said that your researcher’s password protected databases _are_ searchable via the web, if one has certain credentials, it seems to go against the general “spirit” of the deep web. It should also be said that the entire reason the concept was named was not that there were password protected databases out there, but that search engines simply didn’t have the capability of crawling them, whereas if the end user was at the site they certainly could. ---
**Author:** ![Garfield226](https://avatars.discourse-cdn.com/v4/letter/g/9e8a1a/32.png) [@Garfield226](https://boards.straightdope.com/u/Garfield226)\ **Post date:** [June 20, 2004, 8:10pm UTC](https://boards.straightdope.com/t/deep-web-search-engines/251295/9 "2004-06-20T20:10:26Z")
So the specialized databases that you generally have to have a membership to use (usually through a university or business), Lexis-Nexis being the best known (IME), wouldn’t count? That’s what I immediately thought of when **alterego** ’s first reply. ---
**Author:** ![alterego](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@alterego](https://boards.straightdope.com/u/alterego)\ **Post date:** [June 20, 2004, 8:13pm UTC](https://boards.straightdope.com/t/deep-web-search-engines/251295/10 "2004-06-20T20:13:51Z")
> [@Garfield226](#): > > So the specialized databases that you generally have to have a membership to use (usually through a university or business), Lexis-Nexis being the best known (IME), wouldn’t count? > > That’s what I immediately thought of when **alterego** ’s first reply. I personally don’t think they apply. I have access to hundreds of databases online through my university but they aren’t public, they are private. Yes they are ‘invisible’ or ‘deep’ but not for the same reasons that the term “Deep Web” was coined. Maybe someone else has some ideas about it. ---
**Author:** ![Chronos](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/chronos/32/134_2.png) [@Chronos](https://boards.straightdope.com/u/Chronos)\ **Post date:** [June 20, 2004, 8:23pm UTC](https://boards.straightdope.com/t/deep-web-search-engines/251295/11 "2004-06-20T20:23:45Z")
OK, then, a misunderstanding of terminology. Is there some term for the restricted portion of the Internet? Because it seems to me that that would be even larger than the public deep web. Also, the quote > [@](#): > > Search engines cannot find or create these pages. is perhaps a bit misleading, since the search results pages themselves are dynamically generated pages built from a database, and would thus be part of the deep web. So seach engines can, in fact, create some of the deep web pages. ---
**Author:** ![alterego](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@alterego](https://boards.straightdope.com/u/alterego)\ **Post date:** [June 21, 2004, 8:26am UTC](https://boards.straightdope.com/t/deep-web-search-engines/251295/12 "2004-06-21T08:26:14Z")
> [@Chronos](#): > > Also, the quote is perhaps a bit misleading, since the search results pages themselves are dynamically generated pages built from a database, and would thus be part of the deep web. So seach engines can, in fact, create some of the deep web pages. That’s an interesting idea. Technically speaking the exact pages that Google spits out do not exist in static form. On the other hand they are just dynamic representations of static data that we already have access to. It still sounds like they apply, which probably makes places like [archive.org](http://archive.org) with their petabyte boxes and Google some of the biggest sources of the deep web themselves. “I am my greatest problem” 🙂 I don’t know of a term for what you ask. Probably “private web” vice “public web” would work, though that could get confused with a VPN. ---
**Author:** ![Dewey\_Finn](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/dewey_finn/32/4222_2.png) [@Dewey\_Finn](https://boards.straightdope.com/u/Dewey_Finn)\ **Post date:** [June 21, 2004, 2:49pm UTC](https://boards.straightdope.com/t/deep-web-search-engines/251295/13 "2004-06-21T14:49:27Z")
FYI, there’s an [article](http://www.nytimes.com/2004/06/21/technology/21LIBR.html?hp) (free registration required) about this issue in today’s New York Times. It discusses how even academics tend to use only online sources to do research and how librarians are responding to this.