Mortgage Loans - Xbox Mod Chip - Debt Management - Free Advertising - Mortgage
PDA

View Full Version : Robots.txt

SERPalert
Oct 26th 2005, 5:24 am
Firstly which is correct:

User-agent: googlebot
OR
User-agent: Googlebot

Secondly which is correct?

User-agent: GoogleBot
Disallow: ?page=something

Or?

User-agent: Googlebot
Disallow: /?page=something

Help!
mdvaldosta
Oct 26th 2005, 6:27 am
I always use / but it doesn't matter with robots.txt, it's like designing a web page - ?page=something is relative the the page the folder is in and /?page=something is relative the root. Since robots.txt is in root both would work.

http://www.searchengineworld.com/robots/robots_tutorial.htm

That may be helpful if you want a second opinion.
dj1471
Nov 19th 2005, 9:10 am
Firstly which is correct:

User-agent: googlebot
OR
User-agent: Googlebot

The full user agent string Google are using at the moment is:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

So 'Googlebot' should do it.