A simple text file that stops Google (and other search engines that recognize the file and its commands) from crawling the site, selected pages in the site, or selected file types in the site.
Robots.txt is a text file you put on your web page to tell the search robots which pages you would like not to visit.
Robot.txt is simple text file. If you don't want to crawl some of your web pages by google crawler then you can define it in the robot.txt file. you can generate robot.txt file from the google webmaster tools.
It's exactly what most of the people are telling here.. It guides the search engine spider to not to crawl some specific pages...
Robot.txt file is a way to tell search engine spider that which page to crawl and which page to not crawl, it is one type of simple notepad type text file....
Robot.txt file is used to restrict search engine from accessing pages in our site. Follow this link to Create Robot.txt file
I suppose that it is information on the robots exclusion protocol and how to develop a properly validated robots.txt file.
Robots.text is an important text file from the SEO point of view. This file put restrictions of some web pages and tell the search-engines not to crawl those particular pages on which restrictions applied. It tells search-engines for not crawling restricted web-pages of your websites.
What you are referring as Robots.text is actually written as robot.txt. It is a file that contains instructions for search engines robots or crawlers. Usually robot.txt is used to keep away crawlers from pages they shouldn't index or crawl.