Anyone know of a good (preferably free or reasonably priced) log analyser that will track spiders effectively. I just want to know when a spider last visited and what file it requested on each visit. Ive been doing this with an access database but its awfully slow to process text files.
http://www.robotstats.com/en/ I've had mixed results with this free script. It is sorta klunky, but better than nothing, and it keeps a log in a MySQL database so you can see historic data. If you want to see what it looks like: www.bobafind.com/robotstats/ Code (markup): Obviously I don't want a live link there Also, hunting around I found this: http://www.darrinward.com/googletrax/ The demo site is: http://www.darrinward.com/spydertrax/spyderview.cgi Make sure you select a date on the drop down list - I didn't and I thought that only basic information was listed. Once you select a specific date, it has a lot more details like Date / Time, User Agent, Crawler Remote Host, IP Address, Page. I haven't used it yet but I want to test it out.