Due to some server issues, I have applied <meta name="robots" content="noindex, nofollow"> on my site's homepage and other internal pages. but now i want only google to crawl my site and rest crawlers should not. Do i need to remove this code? or Should I put another <meta name="googlebot" content="index, follow"> right below the code. I also know googlebot have the default behaviour to index and follow but does <meta name="robots" content="noindex, nofollow"> effects googlebot too? I am little confused. Your answer will be appreciable.
As long as you have this inserted: <meta name="robots" content="noindex, nofollow"> No search engine will ever follow your site, that includes Google as well. I think you should get the codes which Bing and Yahoo uses to restrict their crawlers access and dump in and remove the above mentioned noindex meta all together because Google will never index your site if it sees it.
You can also use the robots.txt to control what and which robots can crawl your site: http://www.quickonlinetips.com/arch...arch-engines-from-indexing-your-private-site/ Is there a particular reason why you want to block other major search engines such as Bing and Yahoo?
Many thanks to all!!! My home page gets crawled. Should I use <meta name="msnbot" content="noindex, nofollow"> <meta name="slurp" content="noindex, nofollow"> (currently going on) Due to server load. Unnecessary and bad bots crawling our site. NOW ... other bad bots are still crawling my site. I DONT want them to crawl. I know they do not follow robots.txt commands. (coz i hv done that). So, major issues are: # Load on server due to bad bots. # Some unnecessary bots like web18bot, Mozilla/5.0 are not following robots.txt. # Will blocking individual bots through htaccess be helpful? if yes, then please share the code for it?
I have checked all. Bad bots do not follow robots.txt or meta tags instructions ... I am getting little worried now. Because I am feeling helpless
Oh!... it`s a blunder mistake of you . just repair it. use <meta name="robots" content="index, follow" /> then google will crawl your site.
Hi Friends, Thanks for your valuable comments ... Now, I have removed the <meta name="robots" content="noindex, nofollow"> from homepage... coz server load issue has been resolved upto some instinct ... but bad bots still coming on server although intensity is low... Google is crawling and others are NOT (not exactly but still... ). where as, I have learn a lot from here by sharing valuable comments with you guys
if you used this [<meta name="robots" content="noindex, nofollow">] in home page and inner pages then major search engines ignore your site because you have restrict search engine's bots for indexing your site. So removed this code.
Unfortunately, the bad bots typically do not follow the directives in the robots.txt or meta tags, so the only way I know of to block them is by restricting their IP address.
well dear first of all tell me that why you block other crawlers to crawl your site i think you might be do nothing with your site and just follow all bots.