google spiders webbing around robot tags

Discussion in 'robots.txt' started by klown, Jun 26, 2006.

  1. #1
    i was reading
    the todd malicoat interview
    is google ignoring the robots tags and penalizing sites for duplication even where its told not to index, or is google just ignoring the robot tags?

    I'm rather worried about this as i have a contact us page which is the same but has information for 7 different countries, you click on the country your in and it changes a little bit of information, namely the title and headings, and is otherwise the same, i added no index tags to this page.
     
    klown, Jun 26, 2006 IP
  2. Jean-Luc

    Jean-Luc Peon

    Messages:
    601
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #2
    It looks like a bug. They will probably sort this out rapidly. Google is not more "bug free" than Microsoft or others.

    Nothing to worry about, imo.

    Jean-Luc
     
    Jean-Luc, Jun 27, 2006 IP
  3. klown

    klown Peon

    Messages:
    2,093
    Likes Received:
    115
    Best Answers:
    0
    Trophy Points:
    0
    #3
    seems like its not a bug since they followed the robots tags before, and now they do (according to todd) which means they changed their algo for this.
     
    klown, Jun 27, 2006 IP
  4. MLDesigners

    MLDesigners Peon

    Messages:
    34
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Don't worry, only in rare cases does googlebot not follow the tags, and googlebot not following your restrictions is the least of your worries ;)
     
    MLDesigners, Aug 29, 2006 IP