Advantages of robots.txt

Friends i am new in seo field , I have just read about robots.txt file, but i am so confuse about this. Can anyone tell me what is the robots.txt file, why we need this , what the advantages of this in seo. whats things we mention in that file and how we can create this.

Thanks

 

My knowledge of the robots.txt is minimal but i know that you can block certain urls from being indexed by the search engines.

 

Robots.txt won't have a direct effect in site promotion. It is a plain text file that sits in the root directory of your site and tells various search engine spiders that they shouldn't index all or certain parts of your site.
Robots.txt is beneficial if you have parts of your website that are very similar you can block them from being crawled to avoid being flagged as a spammer.

 

yes robots.txt help us preventing private urls from indexing by various search engines which we donot want i know this but i have to find how to code robots.txt

 

robots.txt can be used for blocking duplicate content in some cases

Cheers,

 

ANy info on HOW you block parts of your site?

 

User-agent: *
Disallow: /cgi-bin/
Disallow: /stats/

 

I think Lovely has explain it all !

I would just like to add few things in that

Robots.txt is nothing but just a simple instructions file for search engine crawlers which guide them which page need to crawl and which to skip.

- for security reason if you need to block few pages to be crawled by any search engine crawler then robots.txt is the best option to do it.

Thanks
Faiz Khan

 

Google "robots.txt" and you will find TONS of information and guides how to use it and what advantages have this file.

 

Same topic is already here in this thread
http://forums.digitalpoint.com/showthread.php?t=721892