Seen Google's Robot.txt?

[misohoni]

Interesting stuff to see who've they've blocked!

http://www.google.com/robots.txt

[LGRComp]

It does not look like they blocked any robots from their sites they just disallowed certain folders. What is kinda funny about it is the fact that they use Allow: which is not part of the robots.txt specification as far as I know, but I could be behind in my reading.

I was right after a little reading. Maybe Google can hire me to fix it for them :D

[minstrel]

You're not behind in your reading - it is a (long ago) recommended robots.txt instruction but is not yet standard.

What is interesting is that Google uses it - I wonder if that means that Googlebot does obey this instruction...