Disallowing All But Certain Spiders

Discussion in 'robots.txt' started by digitalpoint, Apr 23, 2005.

0
  1. #1
    Quick question for anyone that is more familiar with robots.txt than I am.

    Let's say I wanted to only allow a few spiders, but give the ones I do allow a few specific disallows.

    Would this work?

    User-agent: Googlebot
User-agent: Slurp
User-agent: msnbot
User-agent: askjeeves
Disallow: /images/
Disallow: /purchase.php
Disallow: /go.php

User-agent: *
Disallow: /
    Code (markup):
     
    digitalpoint, Apr 23, 2005 IP
  2. fryman

    fryman Kiss my rep

    Messages:
    9,604
    Likes Received:
    777
    Best Answers:
    0
    Trophy Points:
    370
    #2
    I think so, look a lot like the one in the article shere

    http://www.search-marketing.info/meta-tags/robots-meta.htm
     
    fryman, Apr 24, 2005 IP
  3. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #3
    Yeah, as a FYI, it seems to be working... lame bots are not continuing after reading the robots.txt file, but so far only one of the allowed bots has visited (Yahoo), and it continued spidering after reading the robots.txt file.
     
    digitalpoint, Apr 24, 2005 IP
  4. fryman

    fryman Kiss my rep

    Messages:
    9,604
    Likes Received:
    777
    Best Answers:
    0
    Trophy Points:
    370
    #4
    You mean the Googlebot got stuck?
     
    fryman, Apr 24, 2005 IP