Just google robots.txt and you will get plenty of informative results where you can have basic idea about robots.txt
it can make sure pages of your website is not indexed in google, and also block robots looking to scrape content and data from you. it can also help prevent duplicate content from being indexed. google seo robots.txt to see how your seo campaign can benefit from it.
Log on to http://www.google.com/webmasters/ go to: Crawler access -> Generate robots.txt copy & paste into a .txt file, upload it to your main server folder. The rest is just details Cheers !
sorry, my Printscreen function doesn't work, but i think i was pretty clear about what you have to do, if i didn't state your prob here, i will help you
I totally agree to Manish that Google definition is different from one given by Lemints. You can also be right but for that you need to explain it more clearly.
Hey do you know what is "metcafe" i don't know its exact spelling even. I was asked about it somewhere in my interview. Do you have any knowledge about it.
A robots.txt can't prevent any bots doing anything. It's an instruction, and like all instructions, the bots have to be complicit. A bot looking to scrape content wouldn't be coded to follow the instructions found in the robots.txt
the robots.txt file tells the bots what to index, what not to index on your site you can also give specific commands to the bots