Hi! I would like to understand what techniques are used by blog search engines? If they were updated,lets say once in two weeks, the standard crawling (like Google) would work without problem. However, it's not the case. In recent article about Technorati (http://www.economist.com/people/displayStory.cfm?story_id=4484119) it was said that: "Technorati, however, does not crawl blog pages but listens for notifications—“pingsâ€, in the jargon—from the blogs whenever they are updated. This means that Technorati's window into the subculture of blogging is only seconds or, at worst, minutes out of date." Does it mean, that the blog holder or blog portal is subscribed to Technorati, and any update triggers an event? What methods use the other search engines? What about RSS updates? Since all blogs have now RSS, which are updated, is it possible just to track the RSS updates? Thanks Vadim
That's right. Rss or more general blog updates are "different". Well there is a special mechanism available. If you check your logs you will probably see h**p://blo.gs/ping.php . If not you should check the site. You can manually ping it or you can load h**p://ping.blo.gs/?name=example&url=h**p://example.com/. Cooler is using the rpc interface. They run a webservice that you can xml query to let them know. If you use a common blog service it probably works for you. If not you may do it by hand, or let it just be crawled. Hope that helps you out