1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How do you get archiver to READ my robots?

Discussion in 'robots.txt' started by TheSyndicate, Mar 31, 2012.

0
  1. #1
    User-agent: ia_archiver
    Disallow: /

    All ok and correct robot but still archiver is the biggest eating of my BW. How can i make them stop?
     
    TheSyndicate, Mar 31, 2012 IP
  2. anna30

    anna30 Well-Known Member

    Messages:
    280
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    123
    #2
    As you have disallowed ia_archiver so it will not crawler any pages of your site.
     
    anna30, Apr 3, 2012 IP
  3. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #3
    Well the problem is they do anyway, they keep crawling my site like never before.
     
    TheSyndicate, Apr 5, 2012 IP
  4. tiffanywilliams12i2

    tiffanywilliams12i2 Peon

    Messages:
    164
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    have you talked to your host and asked them to hlep?
     
    tiffanywilliams12i2, Apr 10, 2012 IP
  5. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #5
    yeah they said sometimes robots not listen....
     
    TheSyndicate, Apr 12, 2012 IP
  6. pearl17

    pearl17 Banned

    Messages:
    727
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Hello where can i get the robot.txt script
     
    pearl17, Apr 27, 2012 IP
  7. TheSyndicate

    TheSyndicate Prominent Member

    Messages:
    5,410
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    365
    #7
    you make one yourself.
     
    TheSyndicate, Apr 27, 2012 IP