Has Google evolved from "one big crawler" to dozens of little crawlers? Most folks here have seen the results in Google change from day to day or month to month and I am wondering if the reason might not be that Google data centers might not be standing on their own more now, rather than a group of computers and centers which are fed from one central location? This is pure speculation on my part and I have no idea if this is true or possible, however it would explain why most of us get different info from different data centers over a long period of time. Couldn't the algo be inputed into each data center, but the crawl rates be at different time strickly because the cache at each data center was found at different points in time? Eventually, everything will sort itself out, but new sites would certainly be at a disadvantage for months if not years.
I think that google starts creating new SERs in some datacenters and then propagandize it to the rest It's a normal situation
I'm always getting 6-7 or more yahoo spiders on my pages, usually i never have more than 1 google spider.