Why are my tags still being indexed, is causing problems as there is no content on these pages. Here is the selected line in my robots.txt User-agent: * Disallow: /index.php/tag Here is an example of the page that is being indexed: hxtp://www.mysite.com/index.php/tag/blahblah/ What am i doing wrong here?
If the absolute URL was indexed prior to using the robots.txt standard then you need to manually remove them from the Google index. This can be achieved at Google Webmasters Central.
Hey sorry for the delay in the reply but i thought I would wait and see how it progresses incase it had been indexed before my robots.txt update.. but today i've seen a couple more pages have been indexed with the following format: wwww.mysite.com/index.php/tag/blahblah/ wwww.mysite.com/index.php/tag/blahblah2/ Is it something wrong with my robots.txt ? As described earlier i've used this to try and stop the pages being indexed: User-agent: * Disallow: /index.php/tag