The other day I was bored and watched a bunch of matt cutts videos. Anyway, he said something about a "supplemental crawler" that doesn't visit as frequently. As if there are different crawlers, one that checks supplemental pages, one that crawls regular pages... I wonder if they are other ones too. But, does anyone have a theory on whether these come from different IP addresses? Do you think the supplemental crawler, for instance, has it's own cluster/datacenter... or do you think these are just different applications running across all datacenters?
I was not of the opinion that a crawler was based on supplemental or not. I think "supplemental" is decided after the crawl.