Due to some server issues, I have applied <meta name="robots" content="noindex, nofollow"> on my site's homepage and other internal pages. but now i want only google to crawl my site and rest crawlers should not. Do i need to remove this code? or Should I put another <meta name="googlebot" content="index, follow"> right below the code. I also know googlebot have the default behaviour to index and follow but does <meta name="robots" content="noindex, nofollow"> effects googlebot too? I am little confused. Your answer will definitely be appreciable.
Please remove the code & write the tag as index & follow. In your Robot.txt file you can write the code as you are allowing the GoogleBot, not the other.
Hi there, Yes, I agree to remove " No index. No followed" to your homepage and change as mentioned above, By common sense, "No index, no followed " should be more applicable apply to Non-targeted landing page as like Private policy, terms of use, contact, disclaimer, the only exceptional is sitemap. I had experienced Google indexed these on my site when I used "Index, follow", this also because I have been unending learning new things for doing a better IM. However, I will come back to any of my sites if I have time for correcting towards this Have a great day
Thanks for your answer So, Should I inject <meta name="googlebot" content="index, follow"> instead of <meta name="robots" content="noindex, nofollow">. But, if I remove that code from home page then other crawlers will start making load on server. I just want to allow googlebot, no one else. Also, I have read elsewhere that googlebot have default behavior to index and follow. So is this necessary to put <meta name="googlebot" content="index, follow">.
Yes that was the reason - also add your website in webmaster tool that would boost up the whole process
Thanks for the responses. I have removed <meta name="robots" content="noindex, nofollow"> from the homepage and allowed googlebot to crawl by robots.txt. But by homepage did not crawled yet.
It takes time; it doesn't come when we want it to, it comes after finishing it's coffee and newspaper.
Many thanks to all!!! My home page gets crawled. Yes I know, even done that. Thank You NOW ... other bad bots are still crawling my site. I DONT want them to crawl. I know they do not follow robots.txt commands. (coz i hv done that). So, major issues are: # Load on server due to bad bots. # Some unnecessary bots like web18bot, Mozilla/5.0 are not following robots.txt. # Will blocking individual bots through htaccess be helpful? if yes, then please share the code for it?
Now ... I do not have problem with Google. My major issue is to STOP bad bots, those are responsible of making unnecessary load on server. Thank you... But bad bots do not follow these instructions. So ... I am helpless
Hii Friends You remove the tag on "No Index" No Follow" and use the <meta name="robots" content="index, follow" /> tag bcoz this tag help u index ur site page. They allow to Spider n Crawler to crawl the site.
If you want to restrict other crawlers... And they don't obey robots.txt you will have to *block* them with force in .htaccess. Either based on useragent string matching *or* IP address
Yes, You should apply <meta name="googlebot" content="index, follow"> and remove <meta name="robots" content="noindex, nofollow" />. The Google works as per code. To block the other search engines, disallow that search engines through the robots.txt.
Many many bots ignore robots meta tags and robots.txt. If you want not spidered, you had better pword protect your site or limit it by ip address(es).
Hi Friends, Thanks for your valuable comments ... Now, I have removed the <meta name="robots" content="noindex, nofollow"> from homepage... coz server load issue has been resolved upto some instinct ... but bad bots still coming on server although intensity is low... Google is crawling and others are NOT (not exactly but still...). This was the best answer I was waiting for Thanks ... can I know how long these changes get effected?