Which spider do I need to disallow from crawling my site?

Discussion in 'robots.txt' started by cool_dude, Oct 9, 2009.

0
  1. #1
    Hi,
    Recently my blog is being crawled by a lot of spiders namely googlebot,mediapartners,yahoo,bing,sphere,radian6,Spinn3r,TopBlogsInfo,urlfan and many more.

    Is this normal ?
    Is it good or bad?

    Yesterday alone my site has been crawled by around 5500 spiders.I'm worried if this is okay or not.Your inputs in this regard is highly appreciated.

    Thanks in advance!
     
    cool_dude, Oct 9, 2009 IP
  2. AKSIN

    AKSIN Peon

    Messages:
    159
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Its normal and very good.
     
    AKSIN, Oct 9, 2009 IP
  3. HIG-Jerry

    HIG-Jerry Peon

    Messages:
    27
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    There are some bots that aren't so good though that you can block. I have a list of unfriendly spiders somewhere and will post it if I can find it. I don't personally block them though as I have many sites and just don't want the extra burden.
     
    HIG-Jerry, Oct 9, 2009 IP
  4. Gungz

    Gungz Peon

    Messages:
    65
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    As long as the spider is from well-known search engine, I think it's fine.
     
    Gungz, Oct 13, 2009 IP
  5. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #5
    manish.chauhan, Oct 28, 2009 IP