1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How to control when a bot(yahoo,google) can crawl a website

Discussion in 'robots.txt' started by enikram, Mar 18, 2015.

0
  1. #1
    I see yahoo visiting every 20-30 seconds. MSN as often as every 10 seconds.

    Is it possible to control with robots.txt each bot and tell them when to visit.
    While I don't want to control Google bots, bing and yahoo may need to slow down to perhaps once every 2 minutes.

    Then I'd like to really slow down bots I don't care about at all( Baidu,Yandex,etc) so they can be set at once every 10-20 minutes.

    If possible, it would be great to block completely any other spiders
     
    enikram, Mar 18, 2015 IP
  2. superrichguy

    superrichguy Well-Known Member

    Messages:
    2,225
    Likes Received:
    78
    Best Answers:
    0
    Trophy Points:
    195
    #2
    I don't know if it's possible. I know you can restrict them from searching certain files but a time limit I am unaware of.
    But to block certain bits use something like this:

    User-agent: BadBot
    Disallow: /
     
    superrichguy, Mar 18, 2015 IP
  3. Nigel Lew

    Nigel Lew Notable Member

    Messages:
    4,642
    Likes Received:
    405
    Best Answers:
    21
    Trophy Points:
    295
    #3
    Nigel Lew, Mar 18, 2015 IP
  4. enikram

    enikram Active Member

    Messages:
    244
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    60
    #4
    Today only we had 15,000 hits from Unknown bots.

    Is there a way to block ALL the bots, except say yahoo,bing and google?
     
    enikram, Mar 18, 2015 IP
  5. Nigel Lew

    Nigel Lew Notable Member

    Messages:
    4,642
    Likes Received:
    405
    Best Answers:
    21
    Trophy Points:
    295
    #5
    Nigel Lew, Mar 18, 2015 IP