Ok I am rebuilding the sitemap for my online store. We sell over 50,000 items. so we basically could have that many pages in the sitemap. I am wondering if I should include every single item/product in the site map or if I should limit the # of links in the site map? Right now I am using a free tool called site map builder. Does anyone have suggestions on different software? Thanks
If you have any duplicate pages, I would remove them from the sitemap... pages that have the exact same content and meta titles. You might also consider removing pages that have duplicate title tags, like paginated section, department or category pages. My $.02
Yes, when you have a large amount of pages, you should list all of the pages but break them up into different sitemaps, and list them in your robots.txt file. I use the same sitemap.xml generator. Works great...don't forget to set the date, priority, and frequecy.
Google says not to have more than 100 links on a page. You want to include every product in your sitemap - that's what they're for. (no duplicate pages as zeeMissle says) We develop stores as large as yours using Miva Merchant and there's a $20 auto generating sitemap module that can be configured for SEO URLs and number of links per page. This isn't necessarily a plug for Miva Merchant but is a prime example of a full featured success driven shopping cart that you don't have to spend time searching for additional components after the fact. xml-sitemaps.com is great but would take forever to index a site as large as yours ...
Thanks for the advice. I do appreciate it. Speaking of taking forever... Site Map Builder .NET has been running on my pc for about 4 hrs now. It has processed 14,300 urls and says it has 4,781 left... although the urls to process goes up and down so I have no idea how long it will really take. I do hope this helps out search ranking. The Current map has like 4 pages in it... not sure who did that one.
Yeah the $20 sitemap module I mentioned takes about 10 minutes to generate (ie) 5,000 sitemap pages with 100 URLs per page and is automatic. (it cranks) It exports from the database instead of having to go out and look for links like xml-sitemaps.com. And duplicate content isn't an issue either ...
fyi, the default is to search and grab 10 at a time, you can change that in the options. Up to you, but I tend to set mine to 100. Depending on the size of your site, it can take a few mins to weeks. With large sites, I tend to break them down into seperate .xml files. Again, just make sure you list them in your robots.txt file. The sitemap won't help so much in rankings, but it'll help speed up the process of your pages being indexed quicker. Which in turn will start to rank if they are fully optimized.
The below is the actual statement located on sitemaps.org which was created by Search Engines. Sitemaps should be no larger than 10MB (10,485,760 bytes) and can contain a maximum of 50,000 URLs. These limits help to ensure that your web server does not get bogged down serving very large files. This means that if your site contains more than 50,000 URLs or your Sitemap is bigger than 10MB, you must create multiple Sitemap files and use a Sitemap index file. You should use a Sitemap index file even if you have a small site but plan on growing beyond 50,000 URLs or a file size of 10MB. A Sitemap index file can include up to 1,000 Sitemaps and must not exceed 10MB (10,485,760 bytes). You can also use gzip to compress your Sitemaps. If you XML sitemap will surpass any of these limits, even with compression just create multiple XML sitemaps. You can add more than one sitemap: syntax line in the robots.txt and submit multiple sitemps to engines via their webmaster tools.