Does anyone have a general idea of how long it takes Google to add a recent crawl into the cache? Just noticed I got a deep crawl yesterday on one of my sites and waiting to see what's going to be new in the cache. "how many pages of my new directory already got indexed?", "did those problematic pages get lifted out of supplemental, yet again?" These are the types of questions I pose to the bot Gods (google species).
I'd guess this would depend on your PR. I've seen pages from my blog crawled with 5 minutes, and added to the cache within 30 minutes, of posting (I autosend updated sitemaps after each post). Anita
Google sends out its spider to find out new and fresh contents. So, the more fresh contents your site has, the more frequently your site will be visited by spiders. Blogs that are updated constantly are crawled very often. Forum sites like Digital Point are crawled even more frequently. The higher the PR, the more crawling too. Spiders follow links and high PR = more backlinks.
My server stats, from cpanel, showed me that googlebot visited every page on the site. Showed the URL's it visited. Funny thing is though, I added a XML site map and submitted it to G. Since then I noticed that every page of my site has been slowly being dropped out of the index. A few less every few days. I have no Idea why. Started within a few days of submitting the site map. I was having an issue with my host, and was getting a lot of server timeouts for a couple weeks, off and on. Site was not fully down over that time just intermittent disruptions. A bit more than the usual you get on a shared hosting plan. I notice in Google webmaster tools, that G did try to crawl a few pages and hit it at a time when the server was not responding. G shows an error message. Not sure if this is why its dropping pages, or what. I dont know what else to do other than wait for G to sort it out.
Crawls happen almost everyday, at least that what the stats in Google Sitemap say. But to me, cache is a different story altogether. One day you can see your website is 100% cached across various DCs with firmed PageRank, the next day you see "Cached=No" status with zero PageRank.
I have a similar question, i noticed, googlebot is crawling my site for 2 days now, the stats show it crawled about 1000 pages, but it is not reflected in the index.. My site is huge with more than 30000 pages.. is it the reason why the pages are not yet indexed? as it will wait to complete the site or something? same goes with Msnbot.. since google found my site, i did not bother to submit the sitemap even though it is ready.. will submitting the sitemap make a difference?