Hi All, Havn't posted for a while, currently developing an XML Google sitemap generator. I wanted to get some input from fellow webmasters/users. Currently the generator will crawl a website and list every link within an XML file ready for Google. I use XML sitemaps on about sixty websites, but found that there are restrictions on free sitemap generators I've been using. I was toying with the idea of making users link to the site before a sitemap can be generated, to stop people from crawling competitors sites - what do you think about this? do you have any other ideas? Any ideas from people who use sitemap generators would be seriously considered.
You could add a "verify site ownership" feature, similar to the way Google Sitemaps does it. It's a bit of a hassle, but then the users who are building sitemaps are hardly casual surfers, so they probably wouldn't mind a little extra work.
Point taken, but some site owners may not be able to edit their code because they are using cms systems.
I've had some ideas since posting this: For up to 50 links nothing needs to be set up on the target website For 51 - 500 links a comment tag needs to be added to the home page code. Over that I'm still toying with the idea, maybe a link, maybe registration Because this will be an online completely free XML sitemap generator, I don't want users abusing the system. I plan to let users go up to around 5000 links. What do you think?
This might work. As for people scanning the sites that don't belong to them - it's not like they cannot do that already, with all the free bot software out there. As for site dofficulty with ownership - you are right, I've just run into this issue myself - I wanted to submit my blog, the verification process asked me to upload a text file to my server, but the blog provider I am using doesn't let me do that.