I have been working on a new CMS which a settings, Meta tag robots settings : DONT DISPLAY THE ROBOT TAG INDEX , FOLLOW INDEX, NOFOLLOW NOINDEX, FOLLOW NOINDEX , NOFOLLOW These are the options.. What shall I keep as active ??
First of all, if you do not have a <meta name="robots"> element (it's not required) then the search engines assume INDEX FOLLOW. Probably 99.999% of the time, you will want either no <meta name="robots"> element or a <meta name="robots" content="index,follow"> element. The "index" tells the search engines you want them to index the page, and "follow" tells the engines you want them to follow the outbound links on the page (and in Google's case pass Page Rank to the targets of the outbound links). The others are for special cases where you might not want a page on your site to be indexed but you want them to follow the outbound links (NOINDEX,FOLLOW) or you don't want them to index the page AND you don't want them to follow the outbound links (NOINDEX,NOFOLLOW) or you want them to index the page but you don't want them to follow the outbound links (INDEX, NOFOLLOW).
Ummmm NO. Index,follow is best. Do you not want, say, your home page to pass page rank to the other pages it links to on your site? Don't you want pages on your site to get credit for inbound links from other pages on your site? 99.999% of the time INDEX,FOLLOW is the correct choice.
Definitely index,follow. Index: Please mr spider add my content to your search engine. Follow: Please mr spider follow all the links on this page. An noindex,nofollow is the exact opposite. But you DO want to noindex all your duplicate content and nofollow useless links on your page so you funnel link love in the places it is needed.
If you don't want the search engines to count one of your outbound links as an inbound link for the target page, use the rel="nofollow" attribute on the <a> element for the link. For example... <a href="URLofTargetPage" rel="nofollow">This is your link text</a> This tells the search engines don't follow this ONE link and don't pass page rank to the target page. This is recommended if you are linking to a page in a "bad neighborhood" or doing PR sculpting.