Robots.txt is an instructive file for search engines, which is used to control the crawling or indexing of particular WebPages. It requests search engines, not to crawl the specified or confidential web pages mentioned in it, and we need to submit it in webmaster.Is any thing else that i can Assist you with.
Hi Malikaks, Please check this link http://www.mcanerin.com/EN/search-engine/robots-txt.asp If you still does not get it you can ask me. Thanks & Regards