Most sites contain pages that should not be indexed by the search engines. Administrative pages, for example, ABC's "contact" page: "contact.html". There's no need to have it indexed, so we use the robots.txt file to tell the search engine spider (robot) to ignore it. http://www.robotstxt.org/robotstxt.html Here given link will help you a lot, All the information regarding robots.txt is available.