I want to handpick Robots, spiders or crawlers, But How?

Discussion in 'robots.txt' started by Alevoor, Apr 22, 2010.

0
  1. #1
    Of late, I have seen too much activity by unidentified spiders and crawlers which I guess is not worthy (that is, if they aren't helping in terms of traffic building)

    I want to restrict access to a chosen few like Ask, Alexa and the top search engines.
    Please suggest
    Thanks
     
    Alevoor, Apr 22, 2010 IP
  2. Sketch5646

    Sketch5646 Peon

    Messages:
    11
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    i have the same problem, anyone have a solution here ?
     
    Sketch5646, Sep 28, 2010 IP
  3. seoguy5

    seoguy5 Peon

    Messages:
    160
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I think, the IBP tools spider option would be helpful to you. check that on Google. its available for free trial.
     
    seoguy5, Oct 3, 2010 IP