Best Sitemap Generator for Large Sites

Lpe04: How many pages is the website?

 

Anywhere from 10,000 to 40,000 pages or more depending on how I spider it.'

Thanks.

 

Ive heard good things about googles site map generator. Its free and also automatically generates new sitemaps for you when you add pages, change content etc.

The pitfall is that it is still in beta, uses up resources on your server, might be buggy and more then likely the support for it might not be up to snuff.

type in google

google sitemap generator - Google Code

 

ok thanks I'll check it out. Hopefully it doesn't eat up too many resources.

 

Good luck. Post back letting us know the pros and cons of the beta sitemap generator.

 

Ok, if I ever get it working. I have noticed a couple of free download ones too. thanks. Best of luck to you too.

 

Just today played again with that phyton generator. (it seams working fine since 2005 as there is no new release)
It counts 12000 pages pretty fast and the autosubmit option is nice, though when I saw it for the first time it was looking too complicated.

Install free python windows/Linux does not matter and set scheduler to run it as per your need. The whole setup wont take more than 1/2 hour if you follow the step-by-step directions.
You will by pleased by it I am sure.

 

IMO the google one is the fastest and most comprehensive. It also pings google to give the new sitemap which makes it very easy.

If you can not figure it out you can change the settings in it to give you errors. There is two files the config file and the main python script.

I just manipulated the config file and fixed the errors it gave out one by one.

 

Some sitemapper options (at least they are available in A1 Sitemap Generator) to try:

* Have crawler stick to "GET" requests instead of also making "HEAD" requests
(on a small percentage of websites/webservers this makes a huge difference)
* Set up/down simultaneous connections/threads
* Schedule sitemapper to crawl different time of day (i.e. when server load is low)

Notice that some webservers automatically throttle... or worse completely choke when hit hard... So you may need to experiment a little to get max throughput.

Another important thing is how "close" you are to the server...
(Which give installed scripts an advantage, in particular if bandwidth is a problem)

 

Hi bob,

Whats your take on the google sitemap generator? Im thinking about implementing on my server but beta softwares always worry me when system is in production.

 

Its perfectly fine, I don't just use that one I use about 4 or 5 different generators as each gives different results.

The google one though gives the most content and is alot more configurable. It is hardrer to setup than your average php script but once its going there is little to be done to it and it is alot more configurable than the others.

Give it a go and see if you like it for yourself!

 

Thanks bob,

I'll try it out. Ive heard good things about it but havent talked to anyone that has actually been using it in production.

 

If you use wp, you can using xml sitemap generator plugin

 

Good info, thanks. I'm a little hesitant to purchase a full one at the moment. Might give Google's a try.

 

Only problem is, if you ever want to customize WP, it will break the plugin.

 

I also have looked at the online site-map generators and found that they do add their own content in the xml file. But I edit that line out upload the file and everything is okay.. But for a really big site with thousands of pages I wouldn't recommend it, it's just to time consuming looking through all the urls for anything hidden.

 

Really there is no good sitemap generator in the net for LARGE SITE like 2 lacks and above urls... You can only try with any php script with database interface...

that is true

 

I use the paid version of xml-sitemaps.org. It runs very fast and google love it.

 

PHPWebsite seems to only search your server for folders and files based on the readme file I read, maybe not the best solution. Also Google's sitemap generator requires command line access to your server for install, maybe not the best solution for (or available) for some.

Take care.

 

I would just start off with GSiteCrawler at first and crawl the site in entirety until you get all the links fixed and everything looks solid. You will have a baseline sitemap to work with.

THEN append as you go with a text editor by hand or program your CMS to append to the sitemap XML file after you submit and the page goes live. In a multi user editing environment you're going to want to program all that in.