Hey Guys, So I have this website that sells poker chips and right now the category pages might be a cause of concern. So my questions is if it better to disallow the multiple pages in my robots.txt or no? So I have two pages: http://www.mysite.com/poker-chips.html and http://www.mysite.com/poker-chips.html?p=2. Now the second URL is page two with separate products, but with the same meta data and headers. So again my question, is it better to disallow my second page or let it be?
With the scenario provided, you shouldn't have a problem with duplicate penalties so no need to disallow. Ask yourself, is there a valid reason for having the second page? Will someone visiting my site get value from each page if they view both? If you answer yes to both questions, then you should not have a problem. In your scenario, the answer would be yes to both. The reason for the second page is so you can display products that do not appear on the first page. The people visiting your site will get value from each page if they view both because they are seeing different products on each page.
Understood, But I was thinking more about the rankings perspective. For example if google was looking to rank my website for 'poker chips' and it saw both pages, it might get confused with similar headers and meta data about which one to rank and therefore might rank the wrong page.
The "wrong page"? What exactly do you mean by that? Google will rank the page in search results that it thinks is most like what searchers want to see based on their search terms. I guess I am confused as to why there would be a situation where something is the "wrong page". Looking at the linked site in your forum signature, I see navigation to each page from every other page. No matter what page a searcher ends up on, they can easily move around your site. Now, if someone searches specifically for "Nevada Jack Poker Chips" and they are served up a store page in results (not the actual product's page), Google will probably put them on page 2 where those chips appear in the store. That is where they should be sent (in a scenario where the product page was not an option) and I see no reason why you would want them not to see what they are looking for. On the concern of a category match search, such as someone searching specifically for "poker chips", does it matter if someone ends up on page 2 instead of page 1? You have the navigation in place that is very easy to spot and use, along with your sub-categories on the side. I guess I am not seeing an issue if they do end up on a page other than 1. If you really want to increase the chances of someone ending up on page 1, one thing you could do to help is to actually have more poker chips appear on page 1. Page 1 and 2 each have 3 instances of poker chips. Cases are great, but there is a subcategory for that and a category link from the main page to just cases. On page 3, you have an entire page of poker chips, as with page 4. One of those pages is the likeliest candidate for being returned in a "poker chips" search because that is what they display. They aren't displaying 3 poker chips items and 9 poker chip cases. Not just for Google, but for your shoppers, I would recommend you make sure that they actually see poker chips on page 1. If I searched for poker chips (have purchased some online before, I could be a real customer of yours in the future) and ended up on your first page, I would be gone after seeing 3 fairly unimpressive choices for chips and the rest as cases (I want to see poker chips, not poker chip cases). That is how I operate and, like many potential buyers, would not bother to click page 2 rather than just my browser's back button. Based on page 1 results, for someone looking specifically for poker chips, there is no reason to think the rest of the pages offer anything else. Stick those nice poker chips buried on the later pages up on page 1 so visitors see what they came to see and so Google will know that the Poker Chips category is about poker chips and not poker chip cases. On a side note, one thing I did notice that looks like it could be a problem is I see no way to specify a quantity. If I click on Add To Cart for your poker chips from a category page, it opens the product page with the message at the top "Please specify the quantity of product(s)." On the product page however, there is not option to select quantity until I hit the Add To Cart button on the product page. The pop-up is nice, where you can select color and quantity, but the current set up is confusing as I had no clue that is how I got to be able to choose quantity. I sat there looking at the page after receiving the message about needing to select quantity, with no idea how to do it until I just happened to try to click the Add To Cart on the product page (I was actually looking to see if it would just reload the same page with the same message about quantity being required). I would suggest getting that sorted out so you aren't making it harder than it needs to be for someone to get an item into their cart. Would you mind explaining that a bit as you have me confused. How would using or not using a robots.txt harm your system (I assume you mean web server, but even if you mean home pc... )? All robots.txt does is provide "advisement" for web crawlers/spiders visiting your web site.
@mhovingh: Thanks for your input, I see what you are saying. We are surely going to get that fixed ASAP so a customer can order directly from our category level pages. This has come up multiple times and we having been trying to think of the best way to implement it.
hey, The maximum penalty that a site can have is that they can be banned for a few months or for lifetime.