Wildcard usage on User-agent

Discussion in 'robots.txt' started by gravy834, Mar 4, 2011.

0
  1. #1
    Is it possible to block certain user agents using a wildcard e.g.

    User-agent: Sogou*
    Disallow: /

    Would this block all instances of the various Sogou spiders?
     
    gravy834, Mar 4, 2011 IP
  2. ACME Squares

    ACME Squares Peon

    Messages:
    98
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Robots.txt doesn't support that. * isn't a wildcard.

    I'm not sure how your example would be interpreted by SE, but it could be as:
    User-agent: *
    Disallow: /
     
    ACME Squares, Mar 9, 2011 IP
  3. Alan Smith

    Alan Smith Active Member

    Messages:
    1,263
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    78
    #3
    I think there is no facility of blocking with a wild card in the way you want. But if we block a general instance of google bot then all the instance of google will be blocked automatically, we need not need to block every instance manually.
     
    Alan Smith, Mar 11, 2011 IP
  4. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #4
    Yes, Robots would block all the spiders starting from Sogou. Robots.txt supports wildcard.
     
    manish.chauhan, Mar 22, 2011 IP
  5. ACME Squares

    ACME Squares Peon

    Messages:
    98
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #5
    From robotstxt.org:
    Google, Yahoo and Bing have extended the protocol, but don't rely on it.
     
    ACME Squares, Mar 23, 2011 IP