For whatever reason Googlebot has stopped archiving the board content after finding only 10% of it. That is to say, it still archives new threads but no longer archives older threads. I suggest the SDMB install a Google Sitemaps extension for vBulletin in order to help Google find all of the content. In order to maximize the benefits of allowing Google to spider the boards it would be beneficial if the administrator actually logged into Google and tuned the parameters. It would be useful to do this for Yahoo as well, although Google is the principal player here.
Since it’s obvious the board’s search engine will ever work properly, I second this.
What does this even mean? How do you know what Google is or is not indexing? And 10% of what?
Read the next sentence after the one you quoted.
Some of the reason for that may be in the way the board used to be set up. The old links don’t work, anymore.
For example, the link to this thread is: Googlebot stopped spidering the board after it found only 10% of content - About This Message Board - Straight Dope Message Board
Back in the old days, it would’ve been something like:
http://www.straightdope.com/ubb/Forum2/HTML/526216
Spiders hitting those get 404ed.
The query site:boards.straightdope.com has returned roughly the same number of results for months now - around 45,000 threads. When the board was first opened up to spiders Google was getting around 3,000 new threads per day. If it had all of the content it would have indexed at least 500,000 pages, assuming it only got the first page of each thread. It’s not a problem of Google not being able to find the content - it is accessible. But whatever heuristic Googlebot is using for digging through massive archives is stopping it from getting the rest of the SDMB. It needs a pointer.
OK, that makes a lot more sense. I agree that a Sitemap could certainly help in that instance, assuming all those old threads are still available.
This is all stuff that mods and admins cannot do anything about.
I will pass on the information. Unless you hear directly about this from management there’s nothing further that can be done. We cannot compel anyone to do anything about any of this, including acknowledge that they see it.
blah blah blah.