I'm trying to optimize my Robot.txt for my Wordpress Blog and I want to avoid duplicate content. Should I change anything? # BEGIN XML-SITEMAP-PLUGIN Sitemap: http://www.sitename.com/sitemap.xml.gz # END XML-SITEMAP-PLUGIN User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /category Disallow: /tag Disallow: /comments Disallow: /author Disallow: */trackback Disallow: */comments # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: Mediapartners-Google* Disallow: Allow: /* Code (markup):
Really, you should use the Category Killer plugin for Wordpress and use the following perma links... www.site.com/category/post-name-here/ Make sure to only use summaries for your index and category pages, you'll then only need to block the archives and other items you don't want indexed.