Online Schooling - Find jobs - Military Loans - Debt Consolidation - Kamala Harris

PDA

View Full Version : Seen Google's Robot.txt?


misohoni
Oct 13th 2005, 9:54 pm
Interesting stuff to see who've they've blocked!

http://www.google.com/robots.txt

LGRComp
Oct 13th 2005, 10:18 pm
It does not look like they blocked any robots from their sites they just disallowed certain folders. What is kinda funny about it is the fact that they use Allow: which is not part of the robots.txt specification as far as I know, but I could be behind in my reading.

I was right after a little reading. Maybe Google can hire me to fix it for them :D

minstrel
Oct 14th 2005, 6:29 am
You're not behind in your reading - it is a (long ago) recommended robots.txt instruction but is not yet standard.

What is interesting is that Google uses it - I wonder if that means that Googlebot does obey this instruction...