View Single Post
Old 21st April 2008, 09:34 AM   #2 (permalink)
Simon
Dedicated to life!
 
Simon's Avatar
 
Join Date: Jul 2005
Location: 36°38'4.48"N - 4°42'18.52"W
Posts: 2,064
Send a message via MSN to Simon Send a message via Yahoo to Simon Send a message via Skype™ to Simon
You need to try and isolate which of the many Yahoo Slurp bots is using all this bandwidth, as mentioned in the other recent thread the Chinese Yahoo Slurp can probably be blocked without any adverse effect to your site so long as your not marketing to China.

You've said nothing about what your site is or contains, I had a problem with one of mine, where I had forgotten to put limits on a calendar and Google indexed from 1902 until about 2050 before I realised and stopped it. In my case that caused 150 pages of useless calendar data to be Spidered.

Official Route to block spiders is to use the robots.txt, but if this is just a bad bot masquerading as Slurp then this won't help you, in that case you will need to use .htaccess to block the requests.

Simon
__________________
Freelance PHP Programming
__________________
Simon is offline   Reply With Quote