Hey guys I am in a bit of a pickle right now with my XML sitemap generation. Background: My website is over 400,000 pages and growing constantly every day. This is proving to be hard to keep up with when actually tryign to find out exactly how many pages there are. the site is built on code ignitor PHP framework and references to 5 DBs. I have tried to create an xml sitemap via a script on my server that generates URLs from the DB. This is not working really as it breaks every now and then. Its too server intensive and kills the site. I have had my developers try to optimise this by creating delays in the DB queries etc and what not.. there are just too many pages to index. I did a few google searches and found xml-sitemaps.com which is also a script that is installed on the server to do pretty much the same thing, only it doesnt access the DB tables but actually crawls all the pages. This is also not really working as it breaks all the time and never gets to complete the full sitemap generation. By introducing delays in the indexing, it will take more than 5 days to index everything. But I am not able to get it to work continously. It breaks every few hours and I have to restart. Anyways, I am looking into alternative solutions where I can get an external sitemap generation tool to do all of this for me. This way, I don't have to worry about an extra load on the server resources. I would like to have some control over the crawling like introducing delays after indexing every 10 pages so that the site speed is not affected. Any ideas where I can find such a tool / service?
Sitemaps arent really important as long your linking and menu structure is ok, of course it is always handfully to have them 400k pages in one sitemap would cause long loading time what spiders dont like. Split your sitemaps up into categories so you will got several sitemaps, this article might be usefull: clickclack or this one clickclack
Sitemaps can contain a max of 50,000 urls, so youd have to make 8 or so sitemaps.... my site has over 10 million urls and I dont have a problem at all creating them with simple php that gets urls from inside the database, though I do have a decent dedicated server.
I too have a dedicated server but my site isnt the most optimised. Still working on the optimisation but regardless, I still find it hard to have a full generation of the sitemap. Anyways, anyone know of an online sitemap generator that will do that job?
If you read the articles I posted and also Edual mentioned it you can not make one sitemap with 400k urls. You need to make several sitemaps it is the best to have a own tool to generate this. But if your website isnt even optimized, a sitemap isnt your biggest concern (at least thats my opinion).