1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Revist Code?

Discussion in 'All Other Search Engines' started by TwisterMc, Oct 20, 2004.

  1. #1
    It it possible to tell a bot to only spider a directory once a month? I've see meta tags that say revisit in 30 days but bots come daily anyways. So can you tell them when to come and when not to come?
    SEMrush
     
    TwisterMc, Oct 20, 2004 IP
    SEMrush
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,297
    Likes Received:
    2,602
    Best Answers:
    461
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    Nope... unfortunately not. I looked into it about 6 months ago.
     
    digitalpoint, Oct 20, 2004 IP
  3. TwisterMc

    TwisterMc Mac Guru

    Messages:
    972
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Oh and I was hoping. I guess I just have to do it the hard way. :) Thanks
     
    TwisterMc, Oct 20, 2004 IP
  4. sarahk

    sarahk iTamer Staff

    Messages:
    26,131
    Likes Received:
    3,823
    Best Answers:
    108
    Trophy Points:
    665
    #4
    Hi Twister Mc

    What is the problem you are trying to resolve?

    Sarah
     
    sarahk, Oct 20, 2004 IP
  5. mopacfan

    mopacfan Peon

    Messages:
    3,273
    Likes Received:
    164
    Best Answers:
    0
    Trophy Points:
    0
    #5
    What's the "hard way"?
     
    mopacfan, Oct 20, 2004 IP
  6. TwisterMc

    TwisterMc Mac Guru

    Messages:
    972
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Problem. Google and other bots get stuck in my calendar eating up 2+ gigs of bandwidth a few times a month. I only get 5 free gigs which is plenty w/o this issue. However, I appear to have lots more pages when they do go though my calendar so, the hard way, is to remove my robots.txt file until it spiders and then putting it back up once a month. :D
     
    TwisterMc, Oct 20, 2004 IP