I read a figure on-line somewhere recently that gave me pause. That, worldwide, there are currently more than a billion webpages up, but that even the best search engines out there can only give you access to about 16% of them.
Whoa.
If that’s true (and I assume there’s some thoughtful and knowledgeable persons out there who can confirm and/or refute that figure) I was wondering what are some links, search query techniques, alternative search engines that can be used to explore the (presumably) vast uncharted 86% of the internet out there.
get comfortable (snacks, a drink, go ‘potty’ before you nestle in)
think up a subject you wanna look for
make a new folder in your bookmarks/favorites for said subject
start with the search engines (google, dogpile, excite, alta vista). On each one go to the first 20 or so pages that aren’t a store or a message board (unless that’s what you’re looking for.
when you get to those pages, bookmark them then forget about that page and go back.
once you’ve got those first 60-100 pages bookmarked, start going through the pages, keeping the ones that have either A: good info or B: good links.
if it has good info, yeah! if it has good links, go to those links and repeat steps five through seven.
If this doesn’t give you any good links, then I suggest finding a msg board on the subject and asking for sites.
In the old days, engines did send spiders out to try and map the entire web and I believe, Google (for one), still claims that’s the ultimate goal although the task is more difficult and many rely only on submissions.
Nowadays the majority of web pages aren’t submitted to search engines (for adding to their databases) because they don’t have too much merit – home pages, that kind of thing.
Also, because the search engines are competing with each other for your custom, they now employ quality control measures. This, typically, means that if pages on their database aren’t accessed or linked to as frequently as others covering the same subject matter, they are automatically deleted after a time. The engines are getting pretty sophisticated at weeding out the trash and increasing the overall quality.
It’s worth remembering that the majority of web pages now are not in the English language - you can search other languages by using the search engine options.
Finally, search engines and directories are different. For example, Yahoo is a directory – you don’t get a site listed unless it’s been reviewed by a human first (again, quality control). With search engines like Google, you simply submit your site, it gets added and then this new – ish method of weeding out the less popular pages / sites takes over in a kind of Darwinist methodology (i.e. most linked to and accessed pages are kept, others deleted).