Ive been using Gsitecrawler now for some time, but its becomming very cumbersome as my site now has one heck of a lot of pages. It can take 4-6 hours to spider it entirely, and is split into 4 or 5 seperate index files. does anyone know of a tool that is more suited to much larger websites?
Read my Google Sitemap FAQ: http://forums.digitalpoint.com/showthread.php?t=531765 GSitemap is great imo for crawling any website, its fast, free and reliable.
How many pages do your website have? You might want to try filter out some pages? Automate website scanning at nighttime? You can try A1 Sitemap Generator. For very large websites I recommend following the configuration outlined in this article for best results.