Robots.txt

Discussion in 'robots.txt' started by SERPalert, Oct 26, 2005.

0
  1. #1
    Firstly which is correct:

    User-agent: googlebot
    OR
    User-agent: Googlebot

    Secondly which is correct?

    User-agent: GoogleBot
    Disallow: ?page=something

    Or?

    User-agent: Googlebot
    Disallow: /?page=something

    Help!
     
    SERPalert, Oct 26, 2005 IP
  2. mdvaldosta

    mdvaldosta Peon

    Messages:
    4,079
    Likes Received:
    362
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I always use / but it doesn't matter with robots.txt, it's like designing a web page - ?page=something is relative the the page the folder is in and /?page=something is relative the root. Since robots.txt is in root both would work.

    http://www.searchengineworld.com/robots/robots_tutorial.htm

    That may be helpful if you want a second opinion.
     
    mdvaldosta, Oct 26, 2005 IP
  3. dj1471

    dj1471 Well-Known Member

    Messages:
    97
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    116
    #3
    The full user agent string Google are using at the moment is:
    Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

    So 'Googlebot' should do it.
     
    dj1471, Nov 19, 2005 IP