I have summed up all the robot txt info read here http:// forums.digitalpoint (.com) (showthread.php?t=1259401)
Its a method to tell Google for indexing site.. especially used for not indexing ( not crawl) some fold or some pages in the site.. If go to Google webmaster tool get clear picture with example
google.com/support/webmasters/bin/answer.py?hl=en&answer=40360 Basically, it's a file that lives in the root of your web that friendly crawlers use to determine which URLs on your site they should NOT index. By default, if it does not exist they assume any page they can find on your site is available for indexing. NOTE: Bad crawlers will frequently ignore your robots.txt and index whatever they can find.
A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages. (All respectable robots will respect the directives in a robots.txt file, although some may interpret them differently. However, a robots.txt is not enforceable, and some spammers and other troublemakers may ignore it. For this reason, we recommend password protecting confidential information.) You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one). While Google won't crawl or index the content of pages blocked by robots.txt, Google may still index the URLs if Google find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project (www.dmoz.org), can appear in Google search results. In order to use a robots.txt file, you'll need to have access to the root of your domain (if you're not sure, check with your web hoster). If you don't have access to the root of a domain, you can restrict access using the robots meta tag. I hope you can understand the robot.txt file and its use.
Robots.txt file is a set of instructions for visiting robots or spiders that index the content of a site.
I guess this one could help http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360
a simple .txt file made using note pad like editor giving direction to google & other search engines which page to crawl or which not.
Robot. txt tells the search engine which part of your website not to index. It basically cut off pucblic access to places like your website databse, shopping carts and other private information you don't want available to the general public. I am sure you have got very good recommendations earlier, However, i hope this helps further. Br Seo_genius
I don't see any value in a separate file for instructing robots. All you need to do is use the robots META tag on every page that you want to restrict. Since you usually want over 99% of your pages indexed, setting the robots META tag to nofollow or noindex on your restricted pages should not be cumbersome at all. I have a large site (over 4000 pages) and there are only about 200 of those pages needing to be restricted. Adding a simple META tag to those page cured all GOOGLE problems. However, neither a robots.txt nor a META tag will stop rogue bots.