ia_archiver is not listen to Robots.txt can i block there UP?

Discussion in 'robots.txt' started by TheSyndicate, Jul 10, 2012.

0
  1. #1
    ia_archiver is not listen to Robots.txt can i block there UP?

    All my domains have blocked ia_archiver but still they come back even after few month. They simple do not listen to my request not to crawl my sites.:mad:
    What can you do?
     
    TheSyndicate, Jul 10, 2012 IP
  2. lucar898

    lucar898 Peon

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Looks difficult to prohibit ia_archiver
    I think it might ia_archiver not Search engine spiders
     
    lucar898, Aug 4, 2012 IP
  3. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #3
    I block their IP now when i find it
     
    TheSyndicate, Aug 13, 2012 IP
  4. seoguys04

    seoguys04 Greenhorn

    Messages:
    49
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    18
    #4
    "disallow" is just a directive. It is not necessary that robots will follow it. This is even for Google. There are lotsa incidence that ia_archiver denies to follow the robots.txt directive. It is really hard to keep cool although it is hogging your server time and bandwidth but doesn't give you any referral.
     
    seoguys04, Aug 13, 2012 IP
  5. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #5
    Right sometimes they even sink my server and they should be the good guys.
     
    TheSyndicate, Aug 18, 2012 IP
  6. -[z]-

    -[z]- Active Member

    Messages:
    51
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    93
    #6
    Block their user agent in .htaccess
     
    -[z]-, Nov 7, 2012 IP