Hi is there any product/site/tool that can crawl a site and return a plain list of all urls its crawled? I've written a basic one in php but its too slow. thanks
Yep... It was designed for sitemap creation. You can define the depth of the crawl and so on. I can't remember what its called though. Search for sitemap spider generator or something along the lines.
Oh yeh. i tried http://www.sitemapdoc.com/ but its limited to 500. (im trying to crawl several thousand). Ill try some other sitemap tools though.
try: http://www.livepr.info/internal-pages-page-rank.php or http://itarget.info/Sitewide-PageRank-Inspector/getpr.php this gives you the PR also
Not a sitemap generator, but Xenu's Link Sleuth can full crawl a site , you'll find it here : http://home.snafu.de/tilman/xenulink.html