Confirming new evidence of Google's non spidering.

As some of you may know, Google does not like to index complete sites anymore past a certain level.

For example, I have a site that is 3000 pages and Google would only touch 400 of them for indexing. This happened for 6 months until I put a big fat PR 5 link to the middle of my site and then they indexed more pages.

As soon as that PR 5 link was obtained, they indexed 200 more pages within a day.

So...as this is now confirmed.

Google will only index X amount of pages they feel worthy of and that can achieve a page rank.

For example, if you have 3 links inbound to your site, Google says, ok, 3 links is worth 20% of the site or 400 pages.

What did I learn from this??? Build your site wide and not deep, if you are going to build deep, get an inbound link to the middle of your site.

 

Thats pretty interesting but are you sure this is 100% accurate?

 

Could be true. I have had what similar experiences which could be interpreted likewise.

 

I think the higher your PR the further they will go. They key to getting a low PR or high page site fully indexed is to have a sitemap so all your pages can be reached in the fewest number of clicks from your homepage.

 

It doesn't matter much if you build the site deep or wide. If you have 100s of articles, thats thousands of keywords that you will rank high in the SEs for.

I guess it matters if you're primarily concered with page rank. But PR isn't everything.

 

I can not second that - i have sites with almost no pr / backlinks, and more 20.000 pages in the google index.

 

I think that's common knowledge actually. It's like the higher your PR is, the more pages get indexed. After all, isn't that what PR means, Page Ranking, or ranking of websites! The more important your website is, the more pages will be indexed.

 

Hi

I think there is an element of truth in this. Mine is a new domain so I never really expected traffic from Google, yet because any article can be reached in two clicks (an archive page with every article on it) then I guess google indexes most things and I get lots of traffic for less competitive keywords.

Andy

 

Was wondering why my osC site pages were not geting index. This type of stuff sucks. I think all sites should be on equal ground.

 

You won't find any crawler which indexes your entire site unless it's your own.

 

Does having a sitemap.txt that you've told Google about solve this problem? One of my websites was built over years without much forethought, and some of the pages are several layers down.