# Googlebot stopped spidering the board after it found only 10% of content **URL:** \ **Category:** About This Message Board\ **Created:** [July 28, 2009, 11:47pm UTC](https://boards.straightdope.com/t/googlebot-stopped-spidering-the-board-after-it-found-only-10-of-content/504615 "2009-07-28T23:47:58Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![alterego](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@alterego](https://boards.straightdope.com/u/alterego)\ **Post date:** [July 28, 2009, 11:47pm UTC](https://boards.straightdope.com/t/googlebot-stopped-spidering-the-board-after-it-found-only-10-of-content/504615/1 "2009-07-28T23:47:58Z")
For whatever reason Googlebot has stopped archiving the board content after finding only 10% of it. That is to say, it still archives new threads but no longer archives older threads. I suggest the SDMB install a Google Sitemaps extension for vBulletin in order to help Google find all of the content. In order to maximize the benefits of allowing Google to spider the boards it would be beneficial if the administrator actually logged into Google and tuned the parameters. It would be useful to do this for Yahoo as well, although Google is the principal player here. ---
**Author:** ![Revtim](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/revtim/32/1042_2.png) [@Revtim](https://boards.straightdope.com/u/Revtim)\ **Post date:** [July 29, 2009, 12:35am UTC](https://boards.straightdope.com/t/googlebot-stopped-spidering-the-board-after-it-found-only-10-of-content/504615/2 "2009-07-29T00:35:59Z")
Since it’s obvious the board’s search engine will ever work properly, I second this. ---
**Author:** ![friedo](https://avatars.discourse-cdn.com/v4/letter/f/8edcca/32.png) [@friedo](https://boards.straightdope.com/u/friedo)\ **Post date:** [July 29, 2009, 12:37am UTC](https://boards.straightdope.com/t/googlebot-stopped-spidering-the-board-after-it-found-only-10-of-content/504615/3 "2009-07-29T00:37:36Z")
> [@](#): > > For whatever reason Googlebot has stopped archiving the board content after finding only 10% of it. What does this even mean? How do you know what Google is or is not indexing? And 10% of _what_? ---
**Author:** ![Revtim](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/revtim/32/1042_2.png) [@Revtim](https://boards.straightdope.com/u/Revtim)\ **Post date:** [July 29, 2009, 12:44am UTC](https://boards.straightdope.com/t/googlebot-stopped-spidering-the-board-after-it-found-only-10-of-content/504615/4 "2009-07-29T00:44:11Z")
Read the next sentence after the one you quoted. ---
**Author:** ![SoulFrost](https://avatars.discourse-cdn.com/v4/letter/s/f07891/32.png) [@SoulFrost](https://boards.straightdope.com/u/SoulFrost)\ **Post date:** [July 29, 2009, 1:13am UTC](https://boards.straightdope.com/t/googlebot-stopped-spidering-the-board-after-it-found-only-10-of-content/504615/5 "2009-07-29T01:13:45Z")
Some of the reason for that may be in the way the board used to be set up. The old links don’t work, anymore. For example, the link to this thread is: [Googlebot stopped spidering the board after it found only 10% of content - About This Message Board - Straight Dope Message Board](http://boards.straightdope.com/sdmb/showthread.php?t=526216) Back in the old days, it would’ve been something like: [http://www.straightdope.com/ubb/Forum2/HTML/526216](http://www.straightdope.com/ubb/Forum2/HTML/526216) Spiders hitting those get [404ed](http://www.straightdope.com/ubb/Forum7/HTML/000851.html). ---
**Author:** ![alterego](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@alterego](https://boards.straightdope.com/u/alterego)\ **Post date:** [July 29, 2009, 1:30am UTC](https://boards.straightdope.com/t/googlebot-stopped-spidering-the-board-after-it-found-only-10-of-content/504615/6 "2009-07-29T01:30:26Z")
> [@friedo](#): > > What does this even mean? How do you know what Google is or is not indexing? And 10% of _what_? The query site:boards.straightdope.com has returned roughly the same number of results for months now - around 45,000 threads. When the board was first opened up to spiders Google was getting around 3,000 new threads per day. If it had all of the content it would have indexed at least 500,000 pages, assuming it only got the first page of each thread. It’s not a problem of Google not being able to find the content - it is accessible. But whatever heuristic Googlebot is using for digging through massive archives is stopping it from getting the rest of the SDMB. It needs a pointer. ---
**Author:** ![friedo](https://avatars.discourse-cdn.com/v4/letter/f/8edcca/32.png) [@friedo](https://boards.straightdope.com/u/friedo)\ **Post date:** [July 29, 2009, 4:46am UTC](https://boards.straightdope.com/t/googlebot-stopped-spidering-the-board-after-it-found-only-10-of-content/504615/7 "2009-07-29T04:46:18Z")
> [@alterego](#): > > The query site:boards.straightdope.com has returned roughly the same number of results for months now - around 45,000 threads. When the board was first opened up to spiders Google was getting around 3,000 new threads per day. If it had all of the content it would have indexed at least 500,000 pages, assuming it only got the first page of each thread. It’s not a problem of Google not being able to find the content - it is accessible. But whatever heuristic Googlebot is using for digging through massive archives is stopping it from getting the rest of the SDMB. It needs a pointer. OK, that makes a lot more sense. I agree that a Sitemap could certainly help in that instance, assuming all those old threads are still available. ---
**Author:** ![TubaDiva](https://sea3.discourse-cdn.com/straightdope/user_avatar/boards.straightdope.com/tubadiva/32/507_2.png) [@TubaDiva](https://boards.straightdope.com/u/TubaDiva)\ **Post date:** [July 29, 2009, 10:23am UTC](https://boards.straightdope.com/t/googlebot-stopped-spidering-the-board-after-it-found-only-10-of-content/504615/8 "2009-07-29T10:23:43Z")
This is all stuff that mods and admins cannot do anything about. I will pass on the information. Unless you hear directly about this from management there’s nothing further that can be done. We cannot compel anyone to do anything about any of this, including acknowledge that they see it. ---
**Author:** ![alterego](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@alterego](https://boards.straightdope.com/u/alterego)\ **Post date:** [July 29, 2009, 11:09am UTC](https://boards.straightdope.com/t/googlebot-stopped-spidering-the-board-after-it-found-only-10-of-content/504615/9 "2009-07-29T11:09:55Z")
blah blah blah.