hey. i was trying to create good sitemaps, but i needed a tool that let me chosse not to crawl some parts of my website. in exemple. my site displays news i want the news pages to be crawled, but i dont want my website gallery (www.britneyfans.org/gallery) to be crawled, because the gallery has lots of URLS and im just allowed to have about 500. is there any tool that let me create sitemaps excluding some part of the website? thanks
This is the site that i use to make my site maps... it seems to do well for me with making sitemap. http://www.xml-sitemaps.com/
Are you thinking about HTML sitemaps? Because XML sitemaps can be much much much larger than 500 urls. Either way, I will happy to suggest my sitemap creator Here's a list of sitemap creators: http://code.google.com/sm_thirdparty.html
Hi I'm a newbie in this too. Currently I use this: http://www.webflexing.com/ListUrl.php Unlike other generators that produced many duplicate links in an unsorted manner, thus making it difficult to edit, this generator produces sorted links with validator included. The only problem is it crawls only a single web page. Good if you already has an html sitemap for this purpose. Anybody knows of a generator that sort links, eliminate duplicates and crawl the entire site? Thanks in advance. joos.
You can use this tool online sitemap generator to make a standard sitemap for you. And to prevent spiders to crawl special pages of your website, you'd better add the following meta tags into your <head>