The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites. Source: wikipedia.org
HI, In short robots.txt file tells search engines what to do. If you don't want them to download and index your website you can do that here. You can also through in your site maps here so they can easily find them. Most times sites tell Google what they want to index and what they don't want them to index: example: http://forums.digitalpoint.com/robots.txt Notice Digital Points allows them to index some stuff and not they others; usually for SEO purposes. You can read more about robot.txt files here: http://www.robotstxt.org/wc/robots.html Jeff
You might as well not use a robots.txt file... some will tell you "you need a robots.txt" file, but you can run successful sites without such a file. Although: if you want to control which sites to be crawled, which not, then you should use a robots.txt file...
Robots.txt file provides a way to webmasters to tell search engine how web page is treated. This provides a proper access to crawlers about web page indexing and displaying the related page.
It basically tells a search engine what part of your website or which pages on your website (if any) you don't want to be indexed.
Robots.txt file helps search engine robots to direct which part to crawl and which part to avoid. When Search bot or spider of Search Engine comes to your site and wants to index your site, they follow Robots.txt file first. Search bot or spider follows this files direction for index or no index any page of your website.