1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

spider specific and general rules

Discussion in 'robots.txt' started by new, Jan 12, 2006.

  1. #1
    In my robots.txt I have two parts, one for googlebot and the other for general spiders, I want to ask that will the googlebot also look for the rules present in the robots.txt section for all bots or will it only follow the ones defined specifically for it ?

    basically I want to define some global rules and some rules only for google but I also want google to follow the global rules .. I am worried that once I define some rules specifically for googlebot than it will descard the rest of the file
    thanks
     
    new, Jan 12, 2006 IP
  2. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #2
    http://www.robotstxt.org/wc/faq.html#robotstxt

     
    minstrel, Jan 12, 2006 IP
    bogart likes this.
  3. new

    new Peon

    Messages:
    1,433
    Likes Received:
    45
    Best Answers:
    0
    Trophy Points:
    0
    #3
    /\
    But this does not ans my question ?
     
    new, Jan 12, 2006 IP
  4. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #4
    It does if you read it carefully.

    Do this:

    User-agent: *
    Disallow: {insert global rules here}
    
    User-agent: Googlebot
    Disallow: {insert global rules here}
    Disallow: {insert Googlebot specific rules here}
    
    Code (markup):
    Just copy the global rules into the Googlebot section and any extras you need to the Googlebot section.
     
    minstrel, Jan 12, 2006 IP
  5. new

    new Peon

    Messages:
    1,433
    Likes Received:
    45
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Thanks, now it does clarify what I wanted to know :)
     
    new, Jan 12, 2006 IP