Sitemap generator for large sites

Hi,

Just wondering what everyones thaughts are about current site map softwares.

I need a site map generator that can easilly handle more than 200,000 URL's

Currently I am using I architect but when i get to over 17,000 urls it starts to freeze up on me.

So wondering what anyone else can suggest that is good on the current market

 

Try newest version of A1 Sitemap Generator - 1.5.5

At least I know many who scan websites of such sizes without problems. That said, you may still want to configuring options for best memory usage.

 

The free site map generating tools are capable of providing codes to only less than 1000 pages. You have to look for some paid versions and that to you have to design the site map page in such a way that it is free from clutter and look neat. It has to be arranged in the following manner like topic, followed by sub topics and still possible you can go in for third sub division.

 

Hi dairyman
Thanks but i am using a paid sitemap maker by Iarchitect

The only thing i can think of is i need to filter a string or folder or somthing that may be causing my problem but im not exactly sure what.

The problem lies since i have created a huge amount of pages and content with a script called affilistore. http://www.affilistore.com
Which uses mysql database to generate content from datafeeds in affiliate networks.

the section of my site which im still building and in which i use the affilistore product on is http://www.online-shop.com.au/catalogues If anyone is able to review the site if it gives better understanding??? or be able to help me so that my problem is solved that would be great.

 

If you only want to use Iarchitect? Have you tried contacting their support? That is always #1 thing I do when I encounter problems with a product. 200.000 is quite large website (with prior versions of A1SG it could have been a problem as well). Anyways, perhaps you could just filter off http://www.online-shop.com.au/catalogues when scanning? Or do you want those pages included in sitemap?

 

Have contacted the tec support via their forum but very slow to reply

I did particulary want the http://www.online-shop.com.au/catalogues to be included in the site map as this section alone makes up probably 90% of my website.

I think i have found one of the problems and that is that if some one clicks on visit store it gives off a redirect link such as **http://www.online-shop.com.au/catalogues/go.php?proddb=2&l=2056
each link has its own unique code number but the common part of the urls are go.php?proddb=

Obviously it is still picking up these links because it has the http://www.online-shop.com.au/catalogues/ part in the redirect link.

should i put /go.php?proddb= in the filiter or just go.php? or ????

 

I can generate sitemap for you

$50

50% advance
10 days to complete



Regards

Alex

 

A1 Sitemap Genertor, you can test 30 day.

 

Why not make use of Google Sitemap generator?

 

Problem with google sitemap generator is because that is python script, and python is not supported on all websites.

 

are these really work well?

 

i suggest you to try a1 sitemap generator or gsitecrawler for large web sites

 

i hope it works well if it is not i have to do my web site's sitemap by hand.

 

i too have the same problem with my site having more then 20k pages... any expert opinions plz who have tried or used for their sites

 

20k pages (or even 200k) is usually no problem at all. Consider trying A1 Sitemap Generator (newest version 1.5.9). Should you contrary to what I expect encounter any problems, please contact me either through email or Digital Point PM

 

try this one: xml-sitemaps (dot) com
it does up to 500 free and unlimited for $19

Here is the brief summary of its features:

1. It generates any kind of sitemap you require: XML, Text, HTML site maps.
2. It is developed in PHP languages and works with most web-server's configurations
3. Fast and easy installation onto the same host that should be crawled
4. Flexible configuration that allows you to set any sitemap parameters and crawler settings
5. Support of LARGE websites, dividing the sitemap on the parts per 50,000 URLs each and Sitemap Index file according to the Google sitemap protocol.
6. "robots.txt" exclusion protocol is supported
7. GZip compression supported (optional)
8. User-friendly progress indication for sitemap generation in manual mode
9. Allows to setup the cron job to create sitemaps without a user interaction
10. Informs (ping) Google automatically when sitemap generation is complete
11. Website structure analysis feature represents the tree-like layout of your pages distribution within folders
12. The script collects all generated sitemaps details and provides the log changes in it, including the number (and the lists) of added and removed pages.
13. The broken links are detected by the application and reported on the special page, providing also the URLs pages that refer to these bad links.

 

is $19.99 for unlimited installations, or just a single domain?

 

If its for unlimited then really great..

 

i bought it, it doesnt seem to be limited to one site.

which is good, unlimited

 

Try A1 sitemap generator. I tried myself on 100K+ without problem. Most of all they offer 30 days trial so you can try it before buy it.