I have a wordpress site that uses custom permalinks. Basically, I'd like to exclude my post pages using robots.txt while leaving my category pages accessible to crawlers. The URLs look like this - - Category Page URL: http://www.mysite.com/2010/category-name/ - Post Page URL: http://www.mysite.com/2010/category-name/post-name/ The problem is that I have literally thousands of post pages. Is there a way, using regex, to disallow anything with a string following a given category name? Would something like the example below work to exclude the post page URL above while leaving the category page URL crawlable? disallow: /2010/category-name/*/ Any help would be much appreciated!