Not compulsory. The default positions are "index" and "follow", among others. But, for example, if you want to make a page "noindex" and/or "nofollow", you can add a robots meta tag.
Winner! Winner! Chicken Dinner! Jim4767 is absolutely correct. I would only add that the default is index,follow,archive and that ONLY if you want noindex and/or nofollow and/or noarchive do you need to include a meta robots.
Yes, there are other default values — "The default values are now assumed to be INDEX, FOLLOW, ARCHIVE, ODP, SNIPPET and YDIR." (http://www.searchtools.com/robots/r...ID=01d7ec54218f03d381b74e55eedd7156#noarchive). But "index" and "follow" are the two that most webmasters need to know about. And again, they are the default. One does not need to add robots meta tags to make the search engines do the default action.
Jim4767 * View Blog Entries Jim4767 is that for Google crawling? I would want to be crawled on my sub pages.... in case i got mis led sorry
Just don't add robots meta tags prohibiting Google from crawling and indexing. If you have some links to those pages, Google will crawl them sooner or later.
Actually if you Disallow a page with robots.txt and that same page has a meta robots element, the crawlers will never request that page so they never see the meta robots element. You should use one or the other... never both... depending on your desired result. For example if you don't mind them crawling the page and following outbound links from the page but you do NOT want the page to appear in the SERPs, it's best to use the <meta name="robots" content="noindex"> element. This allows them to still follow the outbound links but tells them not to display the page in the SERPs. It tells Google to also remove it from the SERPs if the page is already indexed.
Hi, I think you only need to use meta robots tag on those pages you don't like to be indexed or followed. Of course, you could also setup via robots.txt file. have a nice day,
Good Man - Canonical >>>> I mean here>>>> <meta name="robots" content="noindex"> element ---- should i use that in all my site domain namers???? and blogs to????? ----> im getting traffic from all my blogs and sites --- does that really matter...... ???? hey bro - you about Dan Shculz --- well one and done ---- we just have to better with seo ... lol
Its an optional to Webmaster! these days on different CMS the feature of index and follow comes within the Plugin
Hello Webmasters, Last week my wordpress blog was down for four days. In moz crawl i got 605 banned page issue. The message shows 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag and all my on page seo issues are gone. Now my blog is active. Now I am waiting for next crawl to check whether the issue gone or still live.Any idea to rectify that.