Is it normal that the /robots.txt is the most downloaded item from Server stats? Many of my domains show robots.txt as the most hits?
Yeah, it's normal. I think it's everytime a search bot goes onto your site it re-downloads it just to see if there's any changes. It's the same with me too.
Yes it is because every time a search engine visit your site it download robots.txt very first and then start crawling your sit according to the instructions you have provided in robots.txt.
It's really not much bandwidth. Robots.txt is very small (just text). Unless of course you have a lot of instructions in there? If you updated robots.txt and then it wasn't downloaded again, the bots could be doing something that you don't want them to, so this is why they keep downloading it. I don't know about other bots, but GoogleBot will download it once every 24 hours.
So that just means you get a lot or robots visiting you. Have you checked AWStats or similar? This is good for giving you an idea of bot visits. You could always block any you don't want visiting (using .htaccess, based upon their user-agent).
Yes i get a lot sometimes way much! Even brought the server down sometimes. Most of the bandwidth are from robots not from visitors.
You need to analyse your log files with AWStats, Webalizer or similar, and figure out which bots are visiting a lot or it might not even come under bots, but a particular user-agent (or IP) that is coming up a lot (and a bit unusual, not a normal browser one).
Yeah i have done that and blocked some bad once but mainly it is badu or Google that keep loading the robot.txt.
Google will keep doing it every 24 hours (unless you've used .htaccess to change the robots.txt expiry time). I don't know about Badu, but I would suspect it's the same. That file being requested a few times per day won't be causing a server to go down. There's something else wrong with your site or webhost.
No the robot.txt wont cause the server to go down but it is kind of depressing to see all the bandwith waster for a Robot file. You say you can change this in .htaccess? Can you do serverwide on a dedicated server? What should you set them 1 month? i hardly change the robot.txt.