Why does Yahoo love my website so much?

General Questions
#1

So I monitor the website visitor stats for our website at work - with all the usual caveats about visitors and sessions and hits, but have recently found a report in our logging software (LiveStats.XSP) which separates out the browser visits from the crawler/spider visits, and it surprised me…

Only 53% of visits to the site (so far this month) were from browsers, and Yahoo was top of the crawlers with (and this is what really surprised me) 42% of the visits. Next came MSN with 3% and Google with 0.5%. And this is consistantly the case over the previous months as well - Yahoo’s crawlers visiting our website 10 times as often as any other…

Is this normal?

Grim

#2

For Yahoo, yes.

Their search engine doesn’t so much crawl through a site as it does trample through it - with a crowd. There have been times when over 200 Yahoo spiders were on my site at once.

Add these lines to your robots.txt file:


User-agent: Slurp
Crawl-delay: 120

That reduces the number of hits from once half second (for my site, at least) to once every two minutes. Substitute “120” with the length, in seconds, you feel comfortable with.