Robots file issue

Discussion in 'Search Engine Optimization' started by yugolancer, Jan 31, 2008.

0
  1. #1
    I need a help about the robots.txt .. namely i want to say allow all bots except these three bots but i have no idea how. i.e.
    User-Agent: *
    Disallow:
    this will allow all the robots
    now if i add this:
    User-agent: Zao
    Disallow: /
    will that prevent the Zao bot to crawl my website or when it sees the first lines will start to crawl ignoring the lines below?

    Thanks,
     
    yugolancer, Jan 31, 2008 IP
  2. astup1didiot

    astup1didiot Notable Member

    Messages:
    5,926
    Likes Received:
    270
    Best Answers:
    0
    Trophy Points:
    280
    #2
    User-agent: Zao
    Disallow: /

    That would prevent only the Zao useragent bot from crawling\indexing your site.
     
    astup1didiot, Jan 31, 2008 IP
  3. yugolancer

    yugolancer Well-Known Member

    Messages:
    320
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    110
    #3
    Even if i have allowed all bots (useragents) to crawl my website entirely ???
    Means i want to allow all other bots except few of them?
    Is this correct way?
     
    yugolancer, Jan 31, 2008 IP
  4. astup1didiot

    astup1didiot Notable Member

    Messages:
    5,926
    Likes Received:
    270
    Best Answers:
    0
    Trophy Points:
    280
    #4
    astup1didiot, Jan 31, 2008 IP