Does Yahoo Slurp really support wildcards in robots.txt?

Discussion in 'robots.txt' started by kmike, Mar 1, 2007.

  1. #1
    Or does its syntax differ between Google and Yahoo?
    From their blog, it looks like they are supporting the wildcards in the entries since at least last November:
    http://www.ysearchblog.com/archives/000372.html

    Here's a snippet of robots.txt:
    Disallow: */private.php
    Code (markup):
    And here's one of the many entries which should not be there:
    Note that GoogleBot obeys this and other similar entries, there was no single request for them by GoogleBot for several days.
     
    kmike, Mar 1, 2007 IP
  2. kmike

    kmike Peon

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Ok, looks like Yahoo doesn't like the leading "*". I rewrote the entries like this:
    Disallow: /*/private.php
    Code (markup):
    and Yahoo bot swallowed it in and stopped fetching private.php.
    I can't see why the leading slash shouldn't be caught by "*" wildcard, it must be a bug in Yahoo's robots.txt handling.
     
    kmike, Mar 1, 2007 IP
  3. dhruv37

    dhruv37 Well-Known Member

    Messages:
    3,222
    Likes Received:
    195
    Best Answers:
    0
    Trophy Points:
    165
    #3
    yeah... Yahoo slurp is really support the wildcard entries.
     
    dhruv37, Mar 5, 2007 IP
  4. agnivo007

    agnivo007 Peon

    Messages:
    4,290
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Well, its a rule for robots.txt etries to lead with / as it's the current directory the robots.txt is in.
     
    agnivo007, Mar 5, 2007 IP
  5. kmike

    kmike Peon

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    This "requirement" doesn't make sense to me if the wildcard support is there. And looks like Google thinks so, too.
     
    kmike, Mar 10, 2007 IP