Seen Google's Robot.txt?

Discussion in 'robots.txt' started by misohoni, Oct 13, 2005.

  1. #1
    misohoni, Oct 13, 2005 IP
  2. LGRComp

    LGRComp Well-Known Member

    Messages:
    516
    Likes Received:
    27
    Best Answers:
    0
    Trophy Points:
    195
    #2
    It does not look like they blocked any robots from their sites they just disallowed certain folders. What is kinda funny about it is the fact that they use Allow: which is not part of the robots.txt specification as far as I know, but I could be behind in my reading.

    I was right after a little reading. Maybe Google can hire me to fix it for them :D
     
    LGRComp, Oct 13, 2005 IP
  3. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #3
    You're not behind in your reading - it is a (long ago) recommended robots.txt instruction but is not yet standard.

    What is interesting is that Google uses it - I wonder if that means that Googlebot does obey this instruction...
     
    minstrel, Oct 14, 2005 IP