How to control when a bot(yahoo,google) can crawl a website

Discussion in 'robots.txt' started by enikram, Mar 18, 2015.

0
  1. #1
    I see yahoo visiting every 20-30 seconds. MSN as often as every 10 seconds.

    Is it possible to control with robots.txt each bot and tell them when to visit.
    While I don't want to control Google bots, bing and yahoo may need to slow down to perhaps once every 2 minutes.

    Then I'd like to really slow down bots I don't care about at all( Baidu,Yandex,etc) so they can be set at once every 10-20 minutes.

    If possible, it would be great to block completely any other spiders
     
    enikram, Mar 18, 2015 IP
  2. superrichguy

    superrichguy Well-Known Member

    Messages:
    2,225
    Likes Received:
    78
    Best Answers:
    0
    Trophy Points:
    195
    #2
    I don't know if it's possible. I know you can restrict them from searching certain files but a time limit I am unaware of.
    But to block certain bits use something like this:

    User-agent: BadBot
    Disallow: /
     
    superrichguy, Mar 18, 2015 IP
  3. Nigel Lew

    Nigel Lew Notable Member

    Messages:
    4,642
    Likes Received:
    406
    Best Answers:
    21
    Trophy Points:
    295
    #3
    Nigel Lew, Mar 18, 2015 IP
  4. enikram

    enikram Active Member

    Messages:
    244
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    60
    #4
    Today only we had 15,000 hits from Unknown bots.

    Is there a way to block ALL the bots, except say yahoo,bing and google?
     
    enikram, Mar 18, 2015 IP
  5. Nigel Lew

    Nigel Lew Notable Member

    Messages:
    4,642
    Likes Received:
    406
    Best Answers:
    21
    Trophy Points:
    295
    #5
    Nigel Lew, Mar 18, 2015 IP