Hi, Can we Disallow all Bots and after that Allow only the legitimate Bots like google, bing... Is that can works? Thx
Bad bots will care less what you put in your robots.txt. A better way to block them is via .htaccess (Google "blocking bad bots via .htaccess").
What's the code to put in .htaccess file to block ALL Bots and Trust only a legitimate Bots like Google and Bing for example? Is there any concise and brief code to do so in .htaccess? Thx
I'm not saying a catch all approach is bad, but I would recommend really taking a hard look at the potential types of bots that are good beyond search engine spiders. Proactively disallowing rather than retroactively blocking is smart, but just make sure you're considering and reviewing logs once in awhile bots that crawl for backlinks (moz) among others used for copyright infringement services, those used by CDNs like Cloudflare for its always Online service, downtime bots, etc. Assuming immediately that any bot that isn't Google or Bing could be potentially problematic if you don't at least consider other bots and what they do.