My website has been redesigned : Open Source Development using drupal CMS. By defualt drupal creates all pages as node structure and tags are taken as taxonomy. When I search site: for my website I find static pages as well as pages with node and taxonomy in the URL being crawled. Should I disallow pages with same content but different URLs one contents node in the url or is it fine with Google. I think as the pages are from the same website Google will take as duplicate content. Please suggest, thanks.
Use the canonical tag: http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html This specifies to google your 'preferred version of a page', so you can have more than 1 URL for it.
This is the best alternate of robots.txt to avoid duplicate content penalty. However, this tag is allowed in Google only for yahoo and Bing, you have to block the pages using robots.txt