The robots.txt file is a set of instructions for search engine bots (spiders). it basically tells them what they can, and cannot index.. it's not a "must" file to have in your root directory.. if you don't have any specific use for it, no need to have it.
robots.txt file is one type of file which help in which URL you want follow and which link you want dis follow. Follow code is: check my website and after domain name put this robots.txt you got some links you can easily understand
robot.txt is the text file which tells search engine crawler about that pages of sites that should not be indexed and followed by crawler.. syntax: User-Agent: [Spider name] Disallow: [File Name to be excluded]
its not compulsory but you have robots.txt file so google update pages quickly and you get good result and updates
robot.txt is the scrpting text file If in which apply in the site then search engine crawler not read you site Content and your site link.
Check this link for more information about robots.txt http://www.webconfs.com/what-is-robots-txt-article-12.php
Robots.txt file is a text file that lists the search engine spiders and what they can and cannot index according to various paths found in the list of directories in that file. You can prevent or enable indexing of content in specific directories using the robots.txt file.
check these links for more details www.robotstxt.org & en.wikipedia.org/wiki/Robots_exclusion_standard
robot.txt is file in which we can recommend the google spider..which webpage it follow and which it do not follow..
from robots.txt you can restrict search engine crawler to crawl your web-pages and folders . for this you need to type code in robots.txt and save it at root directory of site.
In robots.txt file you can tell the crawler about those pages which you do not want to index in the google r any other search engine. you can simply add those URLS that you don not want to index.
It is always better have robots.txt on your web server. It is because if spiders check for this file and not found ,it may report it as a broken link (only a chance).So better we should avoid this chance.