How to create robot.txt file ? how to stop indexing files from cgi-bin ?

[poseidon]

Hi,

I am new to the concept as far as robots.txt is concerned. What exactly it does ? How to use it ? Also I don't want search engines to crawl and index my cgi-bin directory. How can I do it.

Some code will be really helpful.

Regards.

[Cristian Mezei]

You should read this (http://www.searchengineworld.com/robots/robots_tutorial.htm).

[Jean-Luc]

Also I don't want search engines to crawl and index my cgi-bin directory. How can I do it.Use this robots.txt :
User-agent: *
Disallow: /cgi-bin/

Jean-Luc

[poseidon]

so what I have to do is just to create a robots.txt file having

User-agent: *
Disallow: /cgi-bin/

isn't it ?

[Jean-Luc]

Exactly. Make sure you upload the robots.txt in the right directory. You have to be able to view it at www.your-site.com/robots.txt.

Jean-Luc

[noiprox]

i have read the tutorial, have a great robots.txt file, but when i do a sitemap generator, it still indexing those pages i want to disallow


User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /cache/
Disallow: /class/
Disallow: /images/
Disallow: /include/
Disallow: /install/
Disallow: /kernel/
Disallow: /language/
Disallow: /templates_c/
Disallow: /themes/
Disallow: /uploads/

this is an example... its called robots.txt and is in the root

any thoughts?