Credit Score - Car Finance - Consolidation Loans - Bad Credit Mortgage - Pirate Costume

PDA

View Full Version : Question Re: Robots.txt


NewComputer
Jun 12th 2004, 9:12 am
I have a question regarding the actual tesxt involved. If you have this:

User-agent: Ihatebots*
Disallow:

and

User-agent: Ihatebots*
Disallow: /

The first one means allow and the second one mean disallow, correct?

Help Desk
Jun 12th 2004, 9:34 am
I belive second means that you are dissallowing indexing of the root or the "/" directory and its sub-directories.

Smyrl
Jun 12th 2004, 9:41 am
The following allows all robots to visit all files because the wildcard "*" specifies all robots.
User-agent: *
Disallow:

This one keeps all robots out.
User-agent: *
Disallow: /

S