Our website have a dynamic sitemap that we submit to Google its a dynamic file it is generated daily and we enter the current date of lastmod and changefreq is daily. However its not a sitemap on our site for Google to find and crawl naturally our sitemap is a submission to Google to crawl, it is not on our site, no link to it from our site and it is just a file for Google and yahoo to use. Each of the sitemap linked from the index sitemap has different numbers of url and all of them do not get indexed completely for example; one sitemap has 2,432 urls, but only 2,165 got indexed sometimes 101 out of 105 or 591 out of 726 it's really random and no pattern. But for some reason, it crawls 100% on some but about 75% on most. My question, How can my dynamic sitemap achieve 100% crawl rate from Google? Any help would be much appreciated, thanks
% crawl rate depends a lot on url structure. I had project before(~8000 pages), first structure was primitive at all - goods.php?id=XXX - something like this = crawl rate ~10% later it was change into seo friendly names - goodcategory/goodname = crawl rate ~60% later it was updating for using rather complex rules for generation url based on group of main keywords = crawl rate ~100%