How to create robot.txt file ? how to stop indexing files from cgi-bin ?

i cam here to ask the same question but found it already and most helpful
can i disallow: filename.php too? or only directories have to disallow

 

You can disallow the pages as well along with directories

 

hello, im new to this stuff too....you can check bigger sites like: http://www.cnn.com/robots.txt for more info.
and what is the difference between "/cgi-bin/" and "/cgi-bin" which is the right one?

 

That depends on how it's being used. For example - if you have a Perl script you want to call, you couldn't call it without typing the full directory i.e. /cgi-bin/. This is because the server needs to know that it is a directory to look in for the xxxxxx.pl / xxxxx.cgi you want to run. Otherwise your call would look like :

/cgi-binxxxxx.pl

When you really want :

/cgi-bin/xxxxx.pl

However - leaving the trailing slash off is sometimes required if you are running a perl script in an server environment that already appends the slash (which is not typically the case). Or for example - you have a call to a perl script from a php script that looks like this :

$script_call='/xxxxx.pl';
if {
$thishappens;
}
echo '/cgi-bin$script_call';

Make sense?