| my experience is that do NOT use robots.txt, which wont work.
I added this to the .htaccess for a few sites and you wont see it again:
RewriteCond %{HTTP_USER_AGENT} slurp [NC]
RewriteRule .* - [F] # sorry you are 'man behaves baddly'!
Note the theres already RewriteEngine On there and the RewriteBase set accordingly (normally webroot /).
I have not seen slurp since...and wont miss it. |