The official site for the Robots.txt is located here: http://www.robotstxt.org/ It explains what it is, how it works, and how to use it to your advantage. It also explains the downsides of the robots.txt file as far as security and other things go. A very good resource for the novice webmaster.