Robots.txt is a text file usually kept in root directory of your website, to tell search engines what to index, how often etc.
this text file is very essential if you want to restrict the search engines to crawl or get the information of the pages of your site.
Sorry but this is not correct. Robots.txt file is actually to tell search engine what not to index. There are certain pages on your site that you don't want to get indexed such as admin pages, login pages, or other internal information. You can just put those url's in Robots.txt file and these pages will not get indexed.
Some consider robots.txt to be more of a security risk than a benefit, as it potentially exposes system structure. Only well behaved spiders take robots.txt into consideration. Malicious and non-complying spiders are abound, which ignore or may take advantage of robots.txt in a nefarious way.
A simple google search will help you find more details about robots.txt and Robot exclusion standard. As stephen082 said, it's used to inform bots adhering to robots exclusion standard about what not to index.
its a file to controll indexing site from google.. in this can specfy to index or non idex the pages on google
It tells the search bots and indexing bots what should be indexed and what is prevented from indexing.l
If you want more about robots.txt, just see on http://seolearningtips.blogspot.com/2008/10/robotstxt.html. I think that this is very useful to all who want to know about robots.txt.
Robots.txt is a text file usually kept in root directory of your website, to tell search engines which page crawl or get the information of the pages of your site.
"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.