How do you get archiver to READ my robots?

Discussion in 'robots.txt' started by TheSyndicate, Mar 31, 2012.

0
  1. #1
    User-agent: ia_archiver
    Disallow: /

    All ok and correct robot but still archiver is the biggest eating of my BW. How can i make them stop?
     
    TheSyndicate, Mar 31, 2012 IP
  2. anna30

    anna30 Well-Known Member

    Messages:
    280
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    123
    #2
    As you have disallowed ia_archiver so it will not crawler any pages of your site.
     
    anna30, Apr 3, 2012 IP
  3. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #3
    Well the problem is they do anyway, they keep crawling my site like never before.
     
    TheSyndicate, Apr 5, 2012 IP
  4. tiffanywilliams12i2

    tiffanywilliams12i2 Peon

    Messages:
    164
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    have you talked to your host and asked them to hlep?
     
    tiffanywilliams12i2, Apr 10, 2012 IP
  5. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #5
    yeah they said sometimes robots not listen....
     
    TheSyndicate, Apr 12, 2012 IP
  6. pearl17

    pearl17 Banned

    Messages:
    727
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Hello where can i get the robot.txt script
     
    pearl17, Apr 27, 2012 IP
  7. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #7
    you make one yourself.
     
    TheSyndicate, Apr 27, 2012 IP