I'm just trying this site http://www.botsense.com/ as I'm not an expert in bots and wouldn't know how to block a bad one. Basically you sign up and it generates a .htaccess file which does the job for the bad bots that ignore robots.txt Anyone got any advice on this or think its a bad thing? I'm just trying it out on one of my sites at present.
The problem with bad bots is much like the spam problem. Bad bots tend to disguise themselves with user agents that look legit. (That site analyzes the user agent to determine if they should block.) Just like spammers use false email addresses, bad bots most often fake their user agent.