I am new on this forum as you can see. I've read this thread a little bit and I realized nobody actually tried to optimized their custom directory service by creating an application that will go through directory content from time to time and detect dead links. Then it will mark links somehow for further investigation. It will save a lot of your time. You can use my idea for free I don't know is it unique idea ot not. I can even write an client application for spying if somebody is intrested.
phpld already has this option in the admin dead links are set to inactive so that you can check them out