1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Seen Google's Robot.txt?

Discussion in 'robots.txt' started by misohoni, Oct 13, 2005.

0
  1. #1
    misohoni, Oct 13, 2005 IP
  2. LGRComp

    LGRComp Well-Known Member

    Messages:
    516
    Likes Received:
    27
    Best Answers:
    0
    Trophy Points:
    195
    #2
    It does not look like they blocked any robots from their sites they just disallowed certain folders. What is kinda funny about it is the fact that they use Allow: which is not part of the robots.txt specification as far as I know, but I could be behind in my reading.

    I was right after a little reading. Maybe Google can hire me to fix it for them :D
     
    LGRComp, Oct 13, 2005 IP
  3. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #3
    You're not behind in your reading - it is a (long ago) recommended robots.txt instruction but is not yet standard.

    What is interesting is that Google uses it - I wonder if that means that Googlebot does obey this instruction...
     
    minstrel, Oct 14, 2005 IP