Question about User-Agent Disallow for a specific crawler

Discussion in 'robots.txt' started by Sean416, Nov 19, 2009.

0
  1. #1
    Hi,

    If I put in say:

    User-Agent: Example-Bot

    Disallow: /

    User-Agent: *
    Disallow: /secret-stuff
    Disallow: /dont-look-here
    etc

    Would Example-Bot be blocked from the entire site while all other bots would only be blocked from the listed directories?

    Thanks!
     
    Sean416, Nov 19, 2009 IP
  2. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #2
    Yes, it will work as you want it to work. In your code, you have blocked the example-bot from the entire website and subsequently, you have allowed all other bots to visit your website except the files you mentioned in disallow:
     
    manish.chauhan, Nov 19, 2009 IP