I'd like to know how works software like seospyglass or open site explorer (seomoz). How do they find links? I mean now that Yahoo! closed Site Explorer and related Apis Google, Yahoo!/Bing have crawlers and big datacenters, but these companies? I have seen that SEO SpyGlass uses Blekko, Icerocket and other small players but they are small companies compared to Google/Bing (with less crawling resources) Thank you
Honestly I don't think it's an impossible task. Unlike Google/Bing they're only crawling the web and counting links. No index building or ranking. It's a BIG task for sure. I'd imagine it can be done with a few servers. I can easily see a company like SEOMOZ building something this out. It's worth a lot to their community.
I believe Spyglass is now pulling information from multiple services. (Instead of just YSE - before it went down.)