How To Disable All Others Bots Except Googlebot, Yahoo! Slurp, and MSNbot?

Discussion in 'robots.txt' started by gio, Feb 20, 2007.

0
  1. #1
    i need help on robots.txt. I want disable all other bots from crawling my site except googlebot, yahoo slurp and msnbot. How do i do that?

    thanks in advance!
     
    gio, Feb 20, 2007 IP
  2. Mxhub

    Mxhub Active Member

    Messages:
    474
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    55
    #2
    Use this for your robots.txt

     
    Mxhub, Feb 22, 2007 IP
  3. gio

    gio Well-Known Member

    Messages:
    2,390
    Likes Received:
    162
    Best Answers:
    0
    Trophy Points:
    195
    #3
    thank you. have you tried that one?
     
    gio, Feb 24, 2007 IP
  4. Mxhub

    Mxhub Active Member

    Messages:
    474
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    55
    #4
    Yes, i use for my sites.
     
    Mxhub, Feb 25, 2007 IP
  5. lifetalk

    lifetalk Well-Known Member

    Messages:
    1,659
    Likes Received:
    44
    Best Answers:
    0
    Trophy Points:
    115
    #5
    So, you are saying that the above text is all that needs to be in the robots.txt file.. that's all,.. nothing else??
     
    lifetalk, Mar 9, 2007 IP
  6. alexus

    alexus Well-Known Member

    Messages:
    746
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    110
    #6
    why would you want to disable others and allow just those three?
     
    alexus, Mar 18, 2007 IP