Revist Code?

Discussion in 'All Other Search Engines' started by TwisterMc, Oct 20, 2004.

0
  1. #1
    It it possible to tell a bot to only spider a directory once a month? I've see meta tags that say revisit in 30 days but bots come daily anyways. So can you tell them when to come and when not to come?
     
    TwisterMc, Oct 20, 2004 IP
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    Nope... unfortunately not. I looked into it about 6 months ago.
     
    digitalpoint, Oct 20, 2004 IP
  3. TwisterMc

    TwisterMc Mac Guru

    Messages:
    972
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Oh and I was hoping. I guess I just have to do it the hard way. :) Thanks
     
    TwisterMc, Oct 20, 2004 IP
  4. sarahk

    sarahk iTamer Staff

    Messages:
    28,813
    Likes Received:
    4,535
    Best Answers:
    123
    Trophy Points:
    665
    #4
    Hi Twister Mc

    What is the problem you are trying to resolve?

    Sarah
     
    sarahk, Oct 20, 2004 IP
  5. mopacfan

    mopacfan Peon

    Messages:
    3,273
    Likes Received:
    164
    Best Answers:
    0
    Trophy Points:
    0
    #5
    What's the "hard way"?
     
    mopacfan, Oct 20, 2004 IP
  6. TwisterMc

    TwisterMc Mac Guru

    Messages:
    972
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Problem. Google and other bots get stuck in my calendar eating up 2+ gigs of bandwidth a few times a month. I only get 5 free gigs which is plenty w/o this issue. However, I appear to have lots more pages when they do go though my calendar so, the hard way, is to remove my robots.txt file until it spiders and then putting it back up once a month. :D
     
    TwisterMc, Oct 20, 2004 IP