Loan - Jordans - Loans - Cheap Car Insurance - Guest house insurance

PDA

View Full Version : Robots.txt correct?


Jayess
Jan 1st 2005, 9:30 am
Is this correct?

User-agent: scooter
Disallow: /
User-agent: lycos
Disallow: /
User-agent: slurp
Disallow: /
User-agent: googlebot
Disallow: /contact.htm
Disallow: /terms.htm
Disallow: /print.php

This is what I WANT it to do:

User-agent: scooter
Disallow: /
#DISALLOW ALTAVISTA FROM THE WHOLE SITE
User-agent: lycos
Disallow: /
#DISALLOW LYCOS FROM THE WHOLE SITE
User-agent: slurp
Disallow: /
#DISALLOW YAHOO FROM THE WHOLE SITE
User-agent: googlebot
Disallow: /contact.htm
#DISALLOW GOOGLE FROM WWW.MYSITE.COM/CONTACT.HTM
Disallow: /terms.htm
#DISALLOW GOOGLE FROM WWW.MYSITE.COM/TERMS.HTM
Disallow: /print.php
#DISALLOW GOOGLE FROM WWW.MYSITE.COM/PRINT.PHP

Is that what it does?

xml
Jan 1st 2005, 9:32 am
Its fine but you can compact it to:

User-agent: scooter
User-agent: lycos
User-agent: slurp
Disallow: /
User-agent: googlebot
Disallow: /contact.htm
Disallow: /terms.htm
Disallow: /print.php

Jayess
Jan 1st 2005, 9:51 am
Its fine but you can compact it to:

User-agent: scooter
User-agent: lycos
User-agent: slurp
Disallow: /
User-agent: googlebot
Disallow: /contact.htm
Disallow: /terms.htm
Disallow: /print.php

Thank you. Happy New Year.