Hey All, I have a personal blog (wordpress) and have decided to disallow indexing of tags and categories through an SEO plugin in the CMS. I have also disallowed any comment pages, any feed pages and basically anything that's not the actual post in my robots.txt. My problem is that for some reason both Google and Yahoo are indexing some of the tag pages as well as some of the feed pages. For example: www.mypersonalblog.com/the-post-name/feed www.mypersonalblog.com/category/the-category/feed I have these all disallowed in my robots.txt file. Is there any other reason these would be indexed as pages? They aren't linked from anywhere, either on my site or anyone else's. Interestingly, Bing is the only one following my instructions... Thanks!
Can you post the robots.txt file here and the URL of your site maybe I can help you to correct things.
Hi dear I m very interested about this topic and want to give u some of the best info from my side........ A robots.txt disallow rule stops spidering - but URLs can be indexed even though the content isn't crawled. The information listed can come from internal or external backlinks, DMOZ etc. However, seeing the feed indexed on a Wordpress site seems odd. Are you seeing these URLs only for a site: operator result - or are they ranking for some kind of regular query...? I hope its helpful 4 u..
google still crawls those links (like it does with no-follow links), and some of them get indexed... for some unknown reason.
I would rather not post personal info on here..maybe a PM? I am just seeing them in the site: operator. These pages have no backlinks other than my blog itself and I don't link them anyways (maybe WordPress does) Thanks!
Sure PM me the details. Comments are alway's indexed and you cannot stop search engines from doing that. I see that all comments have a nofollow so I see no problem with that. If you not want comments get indexed just turn them off when doing a post.