Should i just block Baidu and Yandex?

Discussion in 'robots.txt' started by TheSyndicate, Sep 20, 2012.

0
  1. #1
    Should i just block these two robots they not listen to my robot.txt and they almost bring down my server. Is there any other way to SLOW them down or do i have to block them in firewall or something?
     
    TheSyndicate, Sep 20, 2012 IP
  2. teddyn1

    teddyn1 Peon

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    1
    Trophy Points:
    0
    #2
    Show me your robots.txt to help you :)
     
    teddyn1, Sep 26, 2012 IP
  3. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #3
    Not about the robot they are correct it is about these two robots do not listen to them.
     
    TheSyndicate, Sep 27, 2012 IP
  4. -[z]-

    -[z]- Active Member

    Messages:
    51
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    93
    #4
    block them in your .htaccess file.

    EG: RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]

    This however means that you will need the user agent of the bot or you could block it's IP
     
    -[z]-, Nov 7, 2012 IP