Quote:
Originally Posted by alfo  Has anyone seen ANY benefit from Yahoo Slurp |
I have never been a Yahoo! fan (stupid name, lousy results, garbage sites, rubbish IM client) but it does send me some beneficial traffic, and banning the second most popular search engine is a pretty drastic step for a webmaster.
I suggest starting with robots.txt and waiting a while for them to update their caches - restrict access to anything you don't want in the search results (images directory
etc) and set the crawl delay to something realistic:
Code:
User-agent: Slurp
Disallow: /images
Disallow: /otherstuff
Crawl-delay: 240
Part of the problem is that each instance of the Slurp spider has to download robots.txt and each instance will only accept a 304 (unchanged) response for other content once it has downloaded the full file - and as we know there are a lot of instances crawling around independently.
If all else fails then ban it in .htaccess as a last resort.
...