Good evening, for a couple of months now I am looking for a web spider. However, I couldn't find what I was looking for. Basically, it should be similar to Xenu Link Sleuth, however, it should be programmed in a server-sided language such as PHP. At a minimum I am looking for these features: * crawling a whole list of websites and checking the http header status of all external links * exporting the result (URL+http header code) in a csv or writing it in a mysql database * multi-threading If anyone of you has an idea where I could look or which existing (PHP) script would be easy to tweak I am more than grateful! Thanks a lot. Mark
Take a look at phpDig but be prepared and have a grunty webhost whatever you use This search threw up some other options: http://www.google.com/search?q=php+spider