The Google Bot has scanned and updated its cached version of my home page several times, but has never followed on to any of my real content - site:http// etc on google still returns only one page.. What is google waiting for? I think I have the correct robot settings. Is there a time delay in which Google waits before performing a complete scan of a site?
Once it does index all of the site, you will still be ranked way down in the search results for quite some time - so don't wait around for google to spider the site before you get on with whatever else needs to be done for the site. (maybe look at free directories to submit to).
It has been my experience that the timeframe to spider your site is very dependent on the PR of incoming links, where the incoming links are pointed, and your internal link structure. I have many sites that follow a typical USA->State->City structure. When I launch my sites and want them indexed fast, I point most of my links at the State pages, not the index page. At this point, its almost robotic ... 3-5 Links from PR5-6 homepages of different sites at the homepage of the new site. 2 PR4-5 links from interior pages of 2 diff sites to each of the 51 State pages (100 total inbound links). A typical (almost guaranteed), schedule for googlebot is: Day 1 - Homepage Spidered (In index in 48 hours) Day 3 - State Pages Spidered (In index in 48 hours) Day 3-10 - City Pages Spidered (In index in 48 hours) If my site is less than 2,000 pages in total, then it works like this every time. I have just begun some much, much larger sites and am still experimenting with the best way to get them totally indexed quickly. One of my current sites that I am logging publicly has 53k pages and the hostory of it can be seen here ... http://www.seopark.com/forums/viewtopic-169
I just signed up to host Google Adds, and I they mention something like waiting 48 hours to see traffic go up. I am wondering if hosting Google adds on the site helps the site get spidered faster. I just went live with Google adds last night, so we will see what happens now.
Hosting Google Ads isn't supposed to help you with indexing for the main SERPs. They claim that the spidering for Adsense by Mediabot is completely independant of the indexing by Googlebot.