robots.txt us easy and all sites usually have one and as so often for all important things in web design there is an official web site for robots.txt including a validator http://www.robotstxt.org/
its not necessary to have but if you will have robots.txt then it will be good. In robots.txt you can mention the webpage or file or folder which you don't want to crawl by the Search Engine.
Hi, this file is used for issuing instructions to search engines regarding what they should and should not crawl. It's a good idea to have one if you're interested in driving more traffic to your site. The reason is that some of the bigger search engines may decide to stay away from your site if it doesn't have a robots.txt.