Sitemap tool for extremly large pages?

Hi everyone,

I setup a single webpage to list all ~40,000 items in my database which is setup with mod_rewrite to be SEO friendly. GSiteCrawler says it can't index it because it is too large. Are there any other tools that do not care about the page size?

Thanks!

 

If your site is very large, crawling will take a long time; in these cases I usually generate the sitemap using a script reading directly from the db or recursing the directory and generating the properly rewritten urls for the sitemap.

 

I think that this is the only way to go. I had to write something similar to this for my directory.

 

Try coffeecup sitemapper

 

Check this thread: http://forums.digitalpoint.com/showthread.php?t=54480

 

I also met this scene

 

A script that you create for your site will be the fastest because it won't crawl, it will use the database directly.

 

Or write a custom input plugin for phpSitemapNG - just extract the links and feed them into the system to let it create the sitemap file(s) automatically.
If you have any question regarding that - drop me a message.
Tobias

 

I've recently beta tested a program that would do exactly what you need it to do. Check it out here:
http://www.templatesresource.com/seo/site-map-equalizer.asp

This program does it all.. Gets your entire website as a xml or html sitemap and even lets you know of 404 and other key things that may reflect on how well your website does.. I've used it on a few websites and have been extremely impressed on how well it functions.
Best of luck!

 

DSR - send me the URL (by mail or PM) if you want me to check it. The GSiteCrawler should have no problem with that many URLs (it might take some time though ).

However, if you have it all in the database, it would be MILES easier to just query the database and have it generate the sitemap file from there. You have all the URLs, why go the long way around by crawling your site? You could even just take the query output, wrap the proper URL-stuff around it and save it as a text file (text listing of URLs) and submit that to Google. No magic involved and it saves you a lot of time, bandwidth and CPU resources.

 

The great and easy way to index big site`s link into sitemap.xml is just crawling the link by folders.

Fx. you have a domain, with news & articles..
Then you can crawl the pages in xxx.domain.xxx/news & save that sitemap in a file.
And then crawl xxx.domain.xxxx/articles & and save then in a new sitemap file..

2 sitemap files..

 

I suppose I'm late, but for everyone else. Sitemap Writer Pro creates sitemaps with nearly 6 million URLs.
good luck.