How to create robot.txt file ? how to stop indexing files from cgi-bin ?

[poseidon]

Hi,

I am new to the concept as far as robots.txt is concerned. What exactly it does ? How to use it ? Also I don't want search engines to crawl and index my cgi-bin directory. How can I do it.

Some code will be really helpful.

Regards.

[Cristian Mezei]

You should read this (http://www.searchengineworld.com/robots/robots_tutorial.htm).

[Jean-Luc]

Also I don't want search engines to crawl and index my cgi-bin directory. How can I do it.Use this robots.txt :
User-agent: *
Disallow: /cgi-bin/

Jean-Luc

[poseidon]

so what I have to do is just to create a robots.txt file having

User-agent: *
Disallow: /cgi-bin/

isn't it ?

[Jean-Luc]

Exactly. Make sure you upload the robots.txt in the right directory. You have to be able to view it at www.your-site.com/robots.txt.

Jean-Luc

[noiprox]

i have read the tutorial, have a great robots.txt file, but when i do a sitemap generator, it still indexing those pages i want to disallow


User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /cache/
Disallow: /class/
Disallow: /images/
Disallow: /include/
Disallow: /install/
Disallow: /kernel/
Disallow: /language/
Disallow: /templates_c/
Disallow: /themes/
Disallow: /uploads/

this is an example... its called robots.txt and is in the root

any thoughts?

[GoGlobal]

Yeah,
It's good but which is the most important use in Robots.txt file.

[manish.chauhan]

i have read the tutorial, have a great robots.txt file, but when i do a sitemap generator, it still indexing those pages i want to disallow


User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /cache/
Disallow: /class/
Disallow: /images/
Disallow: /include/
Disallow: /install/
Disallow: /kernel/
Disallow: /language/
Disallow: /templates_c/
Disallow: /themes/
Disallow: /uploads/

this is an example... its called robots.txt and is in the root

any thoughts?

A Sitemap Generator tool doesn't read the robots.txt, it collects all web page urls and put it into a single file, you have to manually remove the web pages from there.

[udayns]

you can tell help from http://www.robotstxt.org/ This will solve your all query.

[manish.chauhan]

you can tell help from http://www.robotstxt.org/ This will solve your all query.

This is crap..:)

[NickR25]

Don't put it in your sitemap; you will get the error in Google Webmaster Tools (if you use it) that there are some URLs in your sitemap being restricted by robots.txt.

[sriraj46]

Don't put it in your sitemap; you will get the error in Google Webmaster Tools (if you use it) that there are some URLs in your sitemap being restricted by robots.txt.
How would one put robots.txt in sitemap. First of all does the robots file be included in the sitemap page. I guess it doesn't.Correct me if i'm wrong

[manish.chauhan]

How would one put robots.txt in sitemap. First of all does the robots file be included in the sitemap page. I guess it doesn't.Correct me if i'm wrong

You can put the sitemap.xml in robots.txt... For more information check
http://www.sitemaps.org/protocol.php#submit_robots

[shailendra]

This is crap..:)

How can you say that this is crap?

[manish.chauhan]

How can you say that this is crap?

just because it only offers general instructions that one can get from anywhere else. However, when it comes to specific points like use of regular expressions, it doesn't provide solid information.

[ggmittal]

How would one put robots.txt in sitemap. First of all does the robots file be included in the sitemap page. I guess it doesn't.Correct me if i'm wrong


yes... i agree.. robots.txt does need to be included in sitemap...

[proson]

when it comes to specific points like use of regular expressions, it doesn't provide solid information.


hi manish I guess you want to exclude certain files?


if you want to exclude certain file why don't you use meta robots on that page instead?

why complicated things when you can do it easily...

[DareDevils]

yes , this is the one
User-agent: *
Disallow: /cgi-bin/

[infomalaya]

Thanks for the tips!

[3drendering]

He Shailendra,

Manish is Absolutely Right.

Why you Oppose him??

Manish is Right

[MrPJH]

i cam here to ask the same question but found it already and most helpful
can i disallow: filename.php too? or only directories have to disallow

[linkdealer]

You can disallow the pages as well along with directories

[kiwin]

hello, im new to this stuff too....you can check bigger sites like: http://www.cnn.com/robots.txt for more info.
and what is the difference between "/cgi-bin/" and "/cgi-bin" which is the right one?

[sgtcory]

.....what is the difference between "/cgi-bin/" and "/cgi-bin" which is the right one?

That depends on how it's being used. For example - if you have a Perl script you want to call, you couldn't call it without typing the full directory i.e. /cgi-bin/. This is because the server needs to know that it is a directory to look in for the xxxxxx.pl / xxxxx.cgi you want to run. Otherwise your call would look like :

/cgi-binxxxxx.pl

When you really want :

/cgi-bin/xxxxx.pl

However - leaving the trailing slash off is sometimes required if you are running a perl script in an server environment that already appends the slash (which is not typically the case). Or for example - you have a call to a perl script from a php script that looks like this :

$script_call='/xxxxx.pl';
if {
$thishappens;
}
echo '/cgi-bin$script_call';

Make sense?