Can you apply robots.txt to the server

Discussion in 'Programming' started by NewComputer, May 31, 2004.

0
  1. #1
    or do they have to be applied within each website? I don't want certain spiders eating any of my bandwidth. Please advise.
     
    NewComputer, May 31, 2004 IP
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    It's per site (not per server), no way to get around that.
     
    digitalpoint, May 31, 2004 IP
  3. vagrant

    vagrant Peon

    Messages:
    2,284
    Likes Received:
    181
    Best Answers:
    0
    Trophy Points:
    0
    #3
    if it is a case of not wanting them on any part of any of the sites at all, then you could possibly ban the spiders, via their ip, from the server
     
    vagrant, May 31, 2004 IP
  4. misohoni

    misohoni Notable Member

    Messages:
    1,717
    Likes Received:
    32
    Best Answers:
    0
    Trophy Points:
    200
    #4
    There's an amazing thread on robots and how to use them somewhere on this site.
     
    misohoni, May 31, 2004 IP