have a read of http://en.wikipedia.org/wiki/Robots_exclusion_standard or http://www.robotstxt.org/ that should tell you all you need to know.
The main use of a robots.txt file is to give robots instructions to what they can crawl and what they should not crawl. This gives you a little more control over the robots. And since this gives you a little more control over the robots, which means you can issue indexing instructions to specific search engines.
But, for what kind of sites robots.txt is useful? What kind of information we should hide from Google robots? Please guide.
It is suggested to have a robots.txt file in a website as there are few files or folders that you don't want to get visible in search engines. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.
as my site content will be from database, the robots.txt directory control access will it effected? how should i set?
Its used when u dnt want some pages to get crawled...however engines can still index those pages if they hav many backlinks with anchor text matching to search querry...an amazing article describes it all here, how Google/ yahoo handles robots.txt
The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information. This is done primarily in order to save the crawler’s time: bots crawl on a budget – if you ensure that it doesn’t waste time on unnecessary content, it will crawl your site deeper and quicker.
Robots.txt is a text file (not html file) which you put on your site to tell the search robots which pages you would like them not to visit. Robots.txt is mandatory for search engines but generally search engines obey what they are asked not to do. The fact that you put a robots.txt file is something like putting a note “Please, do not enter†on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. If you really have sensitive data, it is too naive to rely on robots.txt to protect it from being indexed and displayed in search results.
The robots.txt file is a set of instructions for visiting robots (spiders) that index the content of your web site pages.