This issue came up in this thread. It seems like Google is doing something different in the way it handles the SDMB domain. For example, if I do a Google search on the following string:
I get these results. (The “site:” qualifier tells it to only search that domain.) If I do the same domain search for “Cecil” I only get 2 hits. The same domain searches for “1920”, “1920s”, or “1920’s” (as in “1920’s style raygun”) turn up no results. Searching for “Opal” also turns up no results. Presumably it’s only indexing threads that are visible without a search at the time it happens to crawl the site. But shouldn’t it continue to return and reindex old threads? The 4th result from my “sdmb” search is a thread from 2000 which would seem to indicate that it does continue indexing old threads. But wouldn’t that guarantee that common things like “1920” and “Opal” would turn up something?
Do the URLs of old threads change? Does SDMB use norobots.txt to prevent crawling of the entire site (presumably to save the hamsters)? If either of those is what’s occuring, then why does the thread from 2000 show up?