Although I'm far from a guru at these things, I have been using robots for years. Here's my dilemma... I started an Amazon affiliate store and of course all of the product pages are duplicate content (product reviews, descriptions, etc). Because I have an exact match domain for what I want to target, I think it would be best to block all of the product pages to avoid a dup content penalty. The problem is they are not in an individual folder... for example here is a category page, which I don't want to block: domain.com/6_none_none_1_Tools-Supplies.html and here is an individual product page, which I do want to block: domain.com/6_none_B000OBCRP6_Salon-Pro-Hair-Bonding-Glue-1oz-Super-Pack-of-12.html In a situation like this, where both the wanted and unwanted pages are in the same directory, is there a way to block some but not the others? For example, let's say I started all the product pages like this: domain.com/6_none_productB000OBCRP6_Salon-Pro-Hair-Bonding-Glue-1oz-Super-Pack-of-12.html Would it be possible to block all pages that start with "6_none_product" even though they're not in a separate directory? p.s. please realize I'm using an Associate-O-Matic, so since it's encoded, I don't have a whole lot of control in this area other that changing htaccess and robots. Thanks!
Sign up for google webmaster tools, inside there they have a section where you can create robot files. Very easy and you should find enough instruction there on how to do it.