1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Robots.txt

Discussion in 'robots.txt' started by SERPalert, Oct 26, 2005.

0
  1. #1
    Firstly which is correct:

    User-agent: googlebot
    OR
    User-agent: Googlebot

    Secondly which is correct?

    User-agent: GoogleBot
    Disallow: ?page=something

    Or?

    User-agent: Googlebot
    Disallow: /?page=something

    Help!
     
    SERPalert, Oct 26, 2005 IP
  2. mdvaldosta

    mdvaldosta Peon

    Messages:
    4,079
    Likes Received:
    362
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I always use / but it doesn't matter with robots.txt, it's like designing a web page - ?page=something is relative the the page the folder is in and /?page=something is relative the root. Since robots.txt is in root both would work.

    http://www.searchengineworld.com/robots/robots_tutorial.htm

    That may be helpful if you want a second opinion.
     
    mdvaldosta, Oct 26, 2005 IP
  3. dj1471

    dj1471 Well-Known Member

    Messages:
    97
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    116
    #3
    The full user agent string Google are using at the moment is:
    Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

    So 'Googlebot' should do it.
     
    dj1471, Nov 19, 2005 IP