Free Sitemap Program for *LARGE* Sites?

I have a large social website with over 300,000 pages of profile information.

A1 Sitemap Generator has failed me, G-Mapper has failed me, Google Sitemap Maker has failed me.

I'm getting tired of installing programs that disappoint.

Does anyone know of a good sitemap making program for large websites?

Thanks.

 

Which server side language are you using on your website?

 

I think you should develop your own web based script to create sitemap for your site. Remember, taht you must use sitemap index file for 300k+ URLs.

 

I have also PM'ed you, but if you are having trouble getting A1 to scan a website with 300.000 URLs, please feel free to contact support (me, hehe). Usually that amount won't cause a problem with proper configuration, at least not with recent versions.

 

It is always better to get stuffs designed for your self because at least you will get what you exactly desire.

 

sorry for the very delayed response, I had it set up to email me if anyone replied, but it never did.

Anyways... the server is *nix based... and I didn't want to ravage it with a script (I've read some are pretty resource intensive).

And yes, I did try to set up a1 for larger sites, but it would still fail (either by not finishing the crawl, and never generating a sitemap).

 

try the sql version of Gsitecrawler. It is slow, but should do the trick.

 

My developer built the sitemap using the DB instead of a physical crawler.

For instance, if your website URL has a pattern like members.domain.com/username or something similar, just run a script that goes through your DB and write to a file changing the /username part with the actual usernames. We built our 800,000+ linnk sitemap within 15-30 minutes using this method and it's a lot less of a resource hog than the crawler method.