I am trying to use a robot.txt to prevent duplicate content with some bad forum URLS These three URLS all point to the same page http://www.mychemistrytutor.com/for...ormation-is-needed/msg5342/?topicseen#msg5342 http://www.mychemistrytutor.com/forums/general-chemistry/some-information-is-needed/ http://www.mychemistrytutor.com/forums/general-chemistry/some-information-is-needed/msg5342/ I need to create a robot.txt that allows for dynamic generation of the topic title name and msg number but blocks everything after that
No you can't do it as robots.txt is not auto updated. You need to add some instructions every time when you add some new urls in your site...
Wildcard are allowed by a few search engines, including Google. Even though if you try to remove an outdated link providing the URL to your robots.txt file, Google will tell you wildcards are not recognized despite they promote them, have a look http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40367