My url-rewriting component made a mistake (in summary) and generated some wrong urls for my existing content. So google is indexing two times a lot of my pages. And my trffic is dropped to %50. I am thinking to remove those invalid urls from google index. May plan is to add all those invalid urls into robot.txt as excluded urls (that google should not crawl). Do you htink this plan work? Do you know any other way? Thanks.
I would do a 301 redirect for your wrongly created URLs. No point in banning Google if you can just send it to better content instead.
Thanks Monty. your link is very useful. This is possible. What about having more than one 301 redirects to the same page. Can this be a problem? (I mean any penalty from google)
This would not be a problem. Google will eventually drop the pages that were incorrectly indexed. The 301 would be the way to go because any links pointing to the duplicate pages will still be counted.
I had to change an OsCommerce site to wordpress and had to redirect match the whole site to the new one. Had to do multiple pages to a new one for a client as well. I haven't noticed any penalties. I think the search engines would rather find new content and decide for themselves than run into a 404 error that goes nowhere. 301s are honest redirects so they shouldn't cause any penalties.