I think this is the best place for this. Quickly, about 4 months ago I completely changed my URL scheme to remove tons of variables from my URLs. For instance &state=TX&city=Houston.. etc etc.. becomes /TX/Houston. I have 301'd all the old URLs to the new URL scheme. Problem is, in about 4 months, google has not added ONE of the new pages. Every single old page has gone supplemental. Please tell me there is a way to correct this. I want my new pages and new URLs to show up, and my old pages with long variable filled URLs to go away. For your reference : http://www.google.com/search?hl=en&q=site:nddb.net&btnG=Google+Search http://www.nddb.net/ (P.S. if you are starting a site.. START with clean URLs, because it is a total pain to change them after the fact.)
Personally I don't believe your pages being in supplemental results has much to do with your 301 redirect. Looking at the Google search you linked to there are about 301,000 pages Google knows about... How many of those have unique content? How many are the same page over and over again with the exception of a business name or a city/state listing? I looked at a few random pages and grabbed a few sentences that I searched for in Google. I found about six other sites which had the exact same text. Sorry to say I don't think you have a 301 redirect issue, I think you have a duplicate content and auto generated content issue. I've taken sites in the past and implemented site-wide 301 redirects to a better URI. It took Google a month or so to figure everything out but it worked just fine.
Well considering the site, it's going to have information that is listed elsewhere. (city and state, doctor names, definitions.. sure they will be elsewhere. Those things aren't really the point of the site.) But there are unique pages... at least 1000 of them. Also, those 301,000 pages, are from when I had about 52k names in the database and the old URI scheme... now I have a new URI scheme and about 360k entries... so it should be indexing them, at least as supplemental. I've seen the index jump between 52k and 301k quite a few times. But nothing since I added tons of new pages... not even indexing them as supplemental. Does google count the template as duplicate content? Sure, every page is surrounded by the same template, but that should be fairly normal. A lot of pages have no user entries yet... but that is normal too. I mean, I pull business name and city/state listing out of a database, and those are fundamental to navigating the site, so if google has an issue with sites like that, then I guess I'm just SOL? But also consider, Google crawls ~30k pages per day for the last two weeks. Look at yahoo compared to google. Even if my results are ALL SUPPOSED to be supplemental, google is still not updating them, not adding the 301 to the index. That is a problem, regardless of dupe content. It crawls the pages, but they haven't made it to the index. How long ago was it that you moved urls with 301s? Because I have done it on other sites in the past and had it work fine, it should be updating the URLs, even if they are going to stay supplemental. There cached index dates are mostly in March 2006. That's a long time for google not to index a page, and that is right around the 301 redirect. Thanks.
1. Close all pages with little or no content in robots.txt This will help to reindex your really valuable pages. 2. Are you using Google Sitemaps?
sitemaps I find is a bit like a double edged sword, useful info but it always seems to hold me back. It is completely normal for your old pages to go supplemental, that basically means 'delete me' but Google are quite slow at this (there's some supplemental results of pages from 2005 that haven' existed since then!). Going from 52k to 360k pages may of caused a small alarm to of gone off in Google, so they will be slow to index you if this is the case. think of your new pages being temporary sandboxed (not the site itself, just the new pages) Google will only index the new pages as proper pages (not supplemetal) but I think the above applies The thing to do is to sit tight and do that magical word we all hate, wait.
Yes, I use sitemaps, have most of the major links in there but avoided but EVERY link in there. But yea, I had supp pages before I did the 301, but not all of them. Now every single one appear to be supplemental. It's as if it sees the 301 and marked all the old URLs to die, but hasn't done it yet, in about 4-5 months. I guess it's same old, same old... build links and wait. There's some other factors that may have contributed, but yahoo and msn picked up the 301s and followed them just fine. All the new URLs show in those two. I just find it very odd, it may well be a duplicate content filter tflight, I don't disagree with you, I just hate the idea what my site looks MFA or something, because that is really not the intention. I wouldn't even run adsense if I didn't have a laundry list of improvements I need the money to hire a professional programmer/designer to complete. =) Plus the fact that, if I don't list those "empty" pages, then people won't see that they can add content there. They will only be empty until I get enough traffic to fill them with decent content. Catch-22 maybe =)