Social Buttons
  1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Disallow all Bots

Discussion in 'robots.txt' started by dabidovic123, Oct 13, 2019.

0
  1. #1
    Hi,

    Can we Disallow all Bots and after that Allow only the legitimate Bots like google, bing...

    Is that can works?

    Thx
    SEMrush
     
    dabidovic123, Oct 13, 2019 IP
    SEMrush
  2. qwikad.com

    qwikad.com Illustrious Member Affiliate Manager

    Messages:
    6,032
    Likes Received:
    1,147
    Best Answers:
    22
    Trophy Points:
    400
    #2
    Bad bots will care less what you put in your robots.txt. A better way to block them is via .htaccess (Google "blocking bad bots via .htaccess").
     
    qwikad.com, Oct 13, 2019 IP
  3. dabidovic123

    dabidovic123 Peon

    Messages:
    3
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #3
    What's the code to put in .htaccess file to block ALL Bots and Trust only a legitimate Bots like Google and Bing for example?

    Is there any concise and brief code to do so in .htaccess?

    Thx
     
    dabidovic123, Oct 14, 2019 IP
  4. ElscottHavoc

    ElscottHavoc Well-Known Member

    Messages:
    76
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    103
    #4
    I'm not saying a catch all approach is bad, but I would recommend really taking a hard look at the potential types of bots that are good beyond search engine spiders.

    Proactively disallowing rather than retroactively blocking is smart, but just make sure you're considering and reviewing logs once in awhile bots that crawl for backlinks (moz) among others used for copyright infringement services, those used by CDNs like Cloudflare for its always Online service, downtime bots, etc.

    Assuming immediately that any bot that isn't Google or Bing could be potentially problematic if you don't at least consider other bots and what they do.
     
    ElscottHavoc, Oct 14, 2019 IP