We want robot

Discussion in 'Apache' started by Aks, May 8, 2007.

  1. #1
    We want to robot restriction using htaccess, is it porssible?
     
    Aks, May 8, 2007 IP
  2. T0PS3O

    T0PS3O Feel Good PLC

    Messages:
    13,219
    Likes Received:
    777
    Best Answers:
    0
    Trophy Points:
    0
  3. Smyrl

    Smyrl Tomato Republic Staff

    Messages:
    13,740
    Likes Received:
    1,702
    Best Answers:
    78
    Trophy Points:
    510
    #3
    To a point it is. See a robots.txt tutorial. Rogue robots will not obey your robots.txt file, however.
     
    Smyrl, May 8, 2007 IP
    Richie_Ni likes this.
  4. T0PS3O

    T0PS3O Feel Good PLC

    Messages:
    13,219
    Likes Received:
    777
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Hence the OP enquiring about doing it via htaccess ;) Good morning :D

    Smyrl has got a point in the sense that they may be faking User Agents (as oppose to not obeying robots policies), rendering even the htaccess method useless. Then you'd have to block by IP which is a never ending chase.
     
    T0PS3O, May 8, 2007 IP
    Smyrl likes this.
  5. sarahk

    sarahk iTamer Staff

    Messages:
    28,808
    Likes Received:
    4,534
    Best Answers:
    123
    Trophy Points:
    665
    #5
    I had a project running called botspotter years ago. I was astounded at the number of bots that visited. I documented many but we're talking thousands and most checked out as legit.

    You need to balance the performance hit, the headaches of identifying bad bots etc on the harm they actually cause.
     
    sarahk, May 9, 2007 IP
  6. Hostaff

    Hostaff Active Member

    Messages:
    107
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    60
    #6
    Have you ever heard about
    "http://www.kloth.net/internet/bottrap.php"
    Code (markup):
    ?

    Endless lists in .htaccess spell trouble in a shared hosting environment and are pretty ineffective.

    btw: sorry, but I can't use live links yet.
     
    Hostaff, May 22, 2007 IP
    sarahk likes this.
  7. tinkerbox

    tinkerbox Peon

    Messages:
    55
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #7
    You can try NiceStat, it can track bots, ban bots using User Agent or IP.
    Many other features, try demo
     
    tinkerbox, May 25, 2007 IP