I found google's robots.txt(http://www.google.com/robots.txt) has some lines as: Allow: /searchhistory/ Code (markup): But the robots.txt specification(http://www.robotstxt.org/wc/norobots.html) said there should no "Allow" tag in this file, does it mean google is wrong?
Read this http://www.freefind.com/library/howto/robots/ There is a tag Allow in robots specification.. But usually not everyone use this
Thanks, man. I think it can easily write a "white list" by using the "Allow" tag, that's what I'm really want to do, thanks.
There is no Allow: directive in the original robots.txt specification. Google, Yahoo and a few others have provided definitions for an Allow: directive. At this time, there is no agreement on a common definition. This means that, in some situations, there are contradictory interpretations of this tentative Allow: directive. If you use it, you should limit its use to the parts of your robots.txt that are user-agent specific (only applicable to user-agents that understand it and that use the same definition as you do). It is certainly better to avoid it. Jean-Luc