| thank you guys for all the input.
what i said was in the context of comparing to robots.txt, and fighting out of control, overzealous crawling. yes i knew ip/user agent could be changed but once idetified it can be banned reliably in .htaccess, not in robots.txt, which bots, legitimate or otherwise, could choose NOT to read/follow at all.
as mentioned at the beginning of this thread, it is a last resort of tackling yahoo's excessive usage of bandwidth. the site in question shows up no3 on page1 on google but nowhere to be seen on yahoo using the same keywords, although google only uses a fraction of bandwidth consumed by yahoo. in this case, legitimacy & respectfulness are not relevant, if i may say so. |