Hi Everyone, I have a site that is something like a wikipedia for individuals. We pre-generated 5,000,000+ profiles, which resulted in a little over 30,000,000 search-optimzed pages based on a database of individuals listed in the united states. The pages rank well for vanity (individual name) searches. We have submitted a full sitemap to google and other search engines. The majority of our traffic is in fact from google. My question is this: of the 30M pages, only about 500,000 are indexed, and that has taken more than 8 months. What can I do to encourage their bot to scan the rest of the pages?
It's all about getting backlinks. Do everything you can to keep bringing the search engine spiders back to your website frequently. Don't just build links to your homepage though. Disperse them throughout your site. Also if you build up enough trust, Google would be much more willing to index your pages quicker. Perhaps it would have been a little bit better if you had that content gradually generated over time. Now you'll have to be patient as Google gradually trusts you more and indexes more of your pages. A good solid internal linking structure helps too. You don't want many pages that are more than 3 or 4 clicks from your homepage. Another thing that may help is to keep gradually adding more new content.