Please help me in listing the useful bots or user agents.

Discussion in 'robots.txt' started by life31, Jun 15, 2009.

0
  1. #1
    I intend to create a robot.txt file only allowing the usefull bots or user agents and disallowing all others.

    Is this recomended? Will it harm me in any way??

    Please do comment also help me in listing the useful bots which I should allow.

    For example Googlebot, Yahoo Slurp, Alexa (IA Archiver), MSNBot, Ask, MSNBot-media, The web archive (IA Archiver), Google Sitemaps, Netcraft.

    These are the genuine bots I came across in my site logs. Any more to include in the list.

    BTW then how do I specify them in the robot.txt file.
     
    life31, Jun 15, 2009 IP
  2. zero_124

    zero_124 Peon

    Messages:
    20
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I think, leave a blank file robots.txt

    0kb ;)
     
    zero_124, Jun 23, 2009 IP
  3. jacobkell

    jacobkell Well-Known Member

    Messages:
    830
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    128
    #3
    Yes there should be list of useful bots for robots.txt plus their ip adress to prevent block.
     
    jacobkell, Jun 28, 2009 IP
  4. zero_124

    zero_124 Peon

    Messages:
    20
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    lease do comment also help me in listing the useful bots which I should allow.

    For example Googlebot, Yahoo Slurp, Alexa (IA Arc
     
    zero_124, Jul 27, 2010 IP