Yahooo search crawler now supporting wildcards in robots.txt We can now use '*' in robots directives for Yahoo! Slurp to wildcard match a sequence of characters in URL. We can use this symbol in any part of the URL string we provide in the robots directive. You can read more on - http://www.ysearchblog.com/archives/000372.html