| just before this thread going cold, i'd like to say using .htaccess to banish any unwanted bots, ips is the reliable way as i am pleased to say i finally kick out slurp (plus a few others) after failing robots.txt (havent seen them in the past 48 hrs).
bots can change their interpretation about robots.txt without notice and as often as and in a way they see fit, cant they?
they cant do anything to bypass .htaccess. |