http://www.almaden.ibm.com/webfountain/ Click the link and it refers to this page - it explains what the crawler is.
There is another thread on Digital Point that gets into this. http://forums.digitalpoint.com/showthread.php?t=1112&page=4&pp=10