Hey there I have a site I've been working on since it was launched in March. I have only just discovered that due to a bug in the shopping cart software under certain circumstances it puts the meta tag for robots telling them noindex, nofollow. Unfortunately my site was affected and has had that tag on it's pages for all this time. From what I understand that tag tells robots to basically stay away from my site, don't follow any links and don't index any pages. However, when I do a site:search I find that google has 147 of my pages listed. So my question is basically what's going on? How come google knows about 147 of my pages? I'm confused, I've had the noindex, nofollow there for months but still my site seems to have been picked up by google. Is that normal? Is it just my site is know about but no links carry any weight??? thanks
This is quite common and actually desireable in most cases for shopping cart pages. Although I would suggest a minor change to it if you have control over it... Allow me to explain. The <meta name="robots" content="noindex,nofollow"> element ONLY applies to the page it's on... NOT to the entire site. Having that element in the <head> of your shopping cart page tells the search engines not to index the shopping cart page and NOT to pass page rank (PR... Link Juice) to the outbound links on the shopping cart page, and not to follow the outbound links on the shopping cart page to discover and index the pages it links to. Typically most sites don't want their shopping cart pages indexed. It's a bad user experience to search for something at Google, click on a link, and be taken directly to a Shopping Cart page. T The only thing I would consider changing if you have the ability to control this <meta> element is nofollow to follow. So you would end up with <meta name="robots" content="noindex,follow">. This way if your shopping cart page links to other pages on your site then it will continue to pass them page rank. It's very rare that you should use a nofollow in a <meta name="robots"> element. It creates a PR sink... a black hole where PR flows into your shopping cart and simply disappears... never being passed out on any of your outbound links because a "nofollow" in the <meta name="robots"> element says nofollow EVERY outbound link on the page and in most cases your shopping cart page will have outbound links to other pages on your site that you DO want to pass PR to (like you home page). In general, it's now bad to "nofollow" internal links as doing so simply wastes PR that you could be passing other pages on your site. You should still use "nofollow" on outbound links pointed to sites you don't trust 100%. Here's a recent article from Matt Cutts explaining PR, how "nofollow" USED to be available for PR Sculpting, and how "nofollow" no longer works for PR Sculpting.
I'm pretty sure that the tag was on every page of my site - I can't check now coz obviously I've changed it now. I'm not sure I understand tho, surely I would want search engines to index my shopping cart? Why do you suggest keeping the noindex? thanks for replying btw
Maybe you are talking about something different than me when you say "shopping cart page". I think of your check out pages as your "shopping cart pages". And there is typically no reason to have your "checkout pages" rank. If you are refering to your product pages which describe each of your product's (description, price, etc.) as your "shopping cart pages" because it has an "add to cart" button on it then... yeah... most definitely... you want those indexed AND followed.
Yeah I have misunderstood what you meant. But bringing it back to the original question, I still don't understand how google had 147 of my pages indexed when I had the noindex, nofollow on every page of my website?