Hi all, Please assist me to control on unidentified bots. Some bad unidentified bots relentlessly accessing bandwidth of my website. I have disallow all robots by robots.txt except google, yahoo and some others. I also set the 403 error for most known bad bots by .htaccess but still here is no improvement in bandwidth usage prevention. I got tied by doing so many methods to stop them but my all efforts are inefficient. Please suggest me how i can deal with this situation....
robots.txt has no effect on bad robots, because they probably don't read it and definitely don't obey it. Blocking robots by htaccess will not prevent them from trying to get in. You will still see them in your logs. But all they take is a few hundred bytes for a 403, instead of the multiple Ks or MBs they would get if they reached the real page.
If you know what bot it is then why don't you block it with robots.txt? If you don't know it's name then you can allow a number of bots and keep rest block for your website. Add the following instruction to your robots.txt: User Agent (Name of allowed bots like Google Bot) Disallow User Agent * Disallow /