Or does its syntax differ between Google and Yahoo? From their blog, it looks like they are supporting the wildcards in the entries since at least last November: http://www.ysearchblog.com/archives/000372.html Here's a snippet of robots.txt: Disallow: */private.php Code (markup): And here's one of the many entries which should not be there: Note that GoogleBot obeys this and other similar entries, there was no single request for them by GoogleBot for several days.
Ok, looks like Yahoo doesn't like the leading "*". I rewrote the entries like this: Disallow: /*/private.php Code (markup): and Yahoo bot swallowed it in and stopped fetching private.php. I can't see why the leading slash shouldn't be caught by "*" wildcard, it must be a bug in Yahoo's robots.txt handling.
Well, its a rule for robots.txt etries to lead with / as it's the current directory the robots.txt is in.
This "requirement" doesn't make sense to me if the wildcard support is there. And looks like Google thinks so, too.