Hey folks! I run a site that has several thousand user generated pages that I want google to find. A list of ALL these pages is located from a link in index.php. Eg when you click on a link from the homepage (lets call the link it list), it brings up a page with several thousand new links. index.php is indexed on google. list is also indexed on google, so I know that the page containing the list is known by google. However, the third layer, the actual links which are contained in list are NOT picked up by google. The list has been up for a couple of months now so there's been plenty of time for indexing. Can anyone help? Thanks in advance!
What I'm saying is how can I ensure that the indexing actually happens. How can I make sure google actually indexes the page. Does that make sense? Not to find out IF it's been indexed, but how to MAKE it get indexed.
What you need is a better site architecture... Having 1000 links on a single page is a terrible idea. Such a page passes almost NO link juice to each of the pages it links to. It doesn't help the search engines figure out which pages are important and which pages are not. Your site architecture and linking structure of that architecture is VERY important in determining which pages are indexed first. You should try to limit your links per page to no more than 100 if possible... fewer if possible. Google used to stop following links on a page after they found about 100 to keep crawling efficient. They will follow more now, but my guess is at some point they will STILL likely stop following links from a page. If your site can be comparmentalized or sectioned off by theme or topics, you might want to look into converting to something like a theme pyramid site architecture. This type of site architecture does wonders for crawlability, PR flow, and flow of relevance assuming your content is conducive to this type of site architecture.
Ok - that's really valuable info - thank very much also to Canonial for the detailed answer. Just about to read the pyramid site architecture link - looks like a nice article. Spock