1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Should i just block Baidu and Yandex?

Discussion in 'robots.txt' started by TheSyndicate, Sep 20, 2012.

0
  1. #1
    Should i just block these two robots they not listen to my robot.txt and they almost bring down my server. Is there any other way to SLOW them down or do i have to block them in firewall or something?
     
    TheSyndicate, Sep 20, 2012 IP
  2. teddyn1

    teddyn1 Peon

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    1
    Trophy Points:
    0
    #2
    Show me your robots.txt to help you :)
     
    teddyn1, Sep 26, 2012 IP
  3. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #3
    Not about the robot they are correct it is about these two robots do not listen to them.
     
    TheSyndicate, Sep 27, 2012 IP
  4. -[z]-

    -[z]- Active Member

    Messages:
    51
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    93
    #4
    block them in your .htaccess file.

    EG: RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]

    This however means that you will need the user agent of the bot or you could block it's IP
     
    -[z]-, Nov 7, 2012 IP