This is an tough one to explain...I will say just Google it on YouTube..and video will give you an up close look into it..to brief it up..it hold information about your site so you site can be tracked. "Black Seo Guy "Signing Off"
Robots.txt is a file placed on your server to tell the various search engine spiders not to crawl or index certain sections or pages of your site. Source: What is Robots.txt File, Creating and Using Robots.txt
this is especially usefull if you have content that you want to reserve for your returning visitors and not just the random googler
Search Engines send an automated program to your site whose purpose is to index i.e record all pages (or important pages) inside search engine catalogs. The automated program is known as bots or crawler or spider in search engine optimization circuit. To prevent your secret files like members images , files , login , administration from getting indexed for public viewing we direct the bots (in robots.txt) 1.What to index 2.What to avoid indexing 3If your web pages address has been changed You can also prevent some bots or search engine spiders from crawling your site (COntent Scrapping Case) This is how you define - http://www.robotstxt.org/robotstxt.html The file name must be robots.txt and be put in your root folder. It is not a must but is a good practice to have robots.txt..