Social Buttons
  1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Can you apply robots.txt to the server

Discussion in 'Programming' started by NewComputer, May 31, 2004.

0
  1. #1
    or do they have to be applied within each website? I don't want certain spiders eating any of my bandwidth. Please advise.
    SEMrush
     
    NewComputer, May 31, 2004 IP
    SEMrush
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,297
    Likes Received:
    2,602
    Best Answers:
    461
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    It's per site (not per server), no way to get around that.
     
    digitalpoint, May 31, 2004 IP
  3. vagrant

    vagrant Peon

    Messages:
    2,284
    Likes Received:
    181
    Best Answers:
    0
    Trophy Points:
    0
    #3
    if it is a case of not wanting them on any part of any of the sites at all, then you could possibly ban the spiders, via their ip, from the server
     
    vagrant, May 31, 2004 IP
  4. misohoni

    misohoni Notable Member

    Messages:
    1,717
    Likes Received:
    32
    Best Answers:
    0
    Trophy Points:
    200
    #4
    There's an amazing thread on robots and how to use them somewhere on this site.
     
    misohoni, May 31, 2004 IP