Gday all, I use the <meta name="robots" content="noindex"/>tag on specific redirection pages on one of my new sites. I'm pretty sure that the redirection pages will eventually be removed out of the google index, but I thought they were never supposed to make it in there... The GoogleBot seems drawn to those pages like a magnet, and has indexed 3 of them (blank redirection pages) while only indexing 2 pages of content. The GoogleBot also seems to take forever to actually include my content pages - not sure why. Any comments welcome!
Ah yes I have a feeling the nofollow, noindex etc tags dont do exactly what we all think they do. I have had similar experience in what you are saying, the pages didnt rank well at all (unless i specifically searched for the title) but they were definetely in the index. Maybe they simply add them to the index but give them zero weighting
Actually, I forgot to mention that I indeed DID indeed put up a robots.txt file as well as the meta tags to be completely sure that they wouldnt' get indexed. I thought I mentioned that, but I guess I didn't. This is what I put in my robots.txt file: User-agent: * Disallow: /goto/ (all the redirection pages are in this folder, and all those file also have the meta tag I described above) So that's even a bigger question... Why didn't the robots.txt file AND the meta tags not work?
There's no need to use both - just one or the other. Chances are though that your pages in that directory may have been indexed before you blocked access to them. But I'd have to see the site (please do not PM me with the link) in order to make that determination (and even then it would probably be just an educated guess).
That's what I was thinking, but I put up all the site pages in the same upload as the robotx.txt file. There wasn't anything to index before then. (???)