Do you block Majestic bots?

Discussion in 'robots.txt' started by Jeffr2014, Apr 9, 2015.

0
  1. #1
    I was looking into my traffic logs today and noticed excessive number of requests from majestic bot majestic12.co.uk. There were 150+ page requests in less than 30 minutes... They all have this for UA:
    "Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www[dot]majestic12.co.uk/bot.php?+)"

    Anybody else noticed this? To block or not to block?
     
    Jeffr2014, Apr 9, 2015 IP
  2. billzo

    billzo Well-Known Member

    Messages:
    961
    Likes Received:
    278
    Best Answers:
    15
    Trophy Points:
    113
    #2
    Yes, I block Majestic. I also block Brandwatch, which is notorious for hitting your website with 8 or more page requests per second.

    I block any bot that consumes a lot of resources and provides no benefit to me. If I am targeting only American traffic, I will block Yandex and Baidu as well, plus a few more.
     
    billzo, Apr 9, 2015 IP
  3. Jeffr2014

    Jeffr2014 Active Member

    Messages:
    254
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    55
    #3
    Thanks for confirmation re Majestic. I do block Yandex and Baidu, didn't know about Brandwatch - will check now.
     
    Jeffr2014, Apr 10, 2015 IP
  4. TheSHosting

    TheSHosting Member

    Messages:
    24
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    48
    #4
    We had a recent issue with one of our customers with Majestic bot, which was hitting hard and increasing his resource usage. We blocked the bot for his website to get rid of it.
     
    TheSHosting, Apr 10, 2015 IP
  5. neteater

    neteater Well-Known Member

    Messages:
    734
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    105
    #5
    havent checked my site logs, will block it from htaccess. or may be from robots file.
     
    neteater, Feb 20, 2016 IP