How to create robot.txt file ? how to stop indexing files from cgi-bin ?

Hi,

I am new to the concept as far as robots.txt is concerned. What exactly it does ? How to use it ? Also I don't want search engines to crawl and index my cgi-bin directory. How can I do it.

Some code will be really helpful.

Regards.

 

You should read this.

 

Use this robots.txt :

User-agent: * 
Disallow: /cgi-bin/

Code (markup):

Jean-Luc

 

so what I have to do is just to create a robots.txt file having

isn't it ?

 

Exactly. Make sure you upload the robots.txt in the right directory. You have to be able to view it at www.your-site.com/robots.txt.

Jean-Luc

 

i have read the tutorial, have a great robots.txt file, but when i do a sitemap generator, it still indexing those pages i want to disallow

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /cache/
Disallow: /class/
Disallow: /images/
Disallow: /include/
Disallow: /install/
Disallow: /kernel/
Disallow: /language/
Disallow: /templates_c/
Disallow: /themes/
Disallow: /uploads/

Code (markup):

this is an example... its called robots.txt and is in the root

any thoughts?

 

Yeah,
It's good but which is the most important use in Robots.txt file.

 

A Sitemap Generator tool doesn't read the robots.txt, it collects all web page urls and put it into a single file, you have to manually remove the web pages from there.

 

you can tell help from http://www.robotstxt.org/ This will solve your all query.

 

This is crap..

 

Don't put it in your sitemap; you will get the error in Google Webmaster Tools (if you use it) that there are some URLs in your sitemap being restricted by robots.txt.

 

How would one put robots.txt in sitemap. First of all does the robots file be included in the sitemap page. I guess it doesn't.Correct me if i'm wrong

 

You can put the sitemap.xml in robots.txt... For more information check
http://www.sitemaps.org/protocol.php#submit_robots

 

How can you say that this is crap?

 

just because it only offers general instructions that one can get from anywhere else. However, when it comes to specific points like use of regular expressions, it doesn't provide solid information.

 

yes... i agree.. robots.txt does need to be included in sitemap...

 

hi manish I guess you want to exclude certain files?


if you want to exclude certain file why don't you use meta robots on that page instead?

why complicated things when you can do it easily...

 

yes , this is the one

 

Thanks for the tips!

 

He Shailendra,

Manish is Absolutely Right.

Why you Oppose him??

Manish is Right