So…
The amount of bandwidth yahoo is pulling is massive on one of my sites…
The site in question is a forum in the UK which ranks well on yahoo but as yahoo search is only used by 7.72% (hitwise April 2007) the traffic is pretty low…
I’ve posted in forums before asking if the bandwidth is worth the traffic.. Now that might sound crazy but having 50 bots on the site day and night is a little OTT imho ?
Anyway… I like yahoo and the results for the UK really arn’t bad so i’ve given the bot a good telling too..
Basically if you need a way to slow yahoo’s bot down on your site use the following in your robots.txt file.
For example, a robots.txt rule to set a crawl-delay of 5 for Yahoo! Slurp looks like:
User-agent: Slurp
Crawl-delay: 5Â
A shorter delay value of 0.5 would look like:
User-agent: Slurp
Crawl-delay: 0.5
Â
More info here:
http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-03.html
Â
Â
