Why site map at all?

I wonder why bother with the sitemap, when google lists the pages that are good enough anyway.

 

I have been asking myself the same question of late. I haven't been bothering to update my sitemaps of late.

It probably makes things easier for Google. You can also suggest crawl rates which may save you some bandwidth.

 

Its a good question - I don't really bother and it does not seem to harm my websites

 

Google and UCLA have recently published a study of 500,000,000 URLs found both by natural discovery and sitemap methods. Of those URLs, Google found 77% by using a sitemap BEFORE they found them naturally.

The answer to why you should use a sitemap is faster indexing.

http://www2009.eprints.org/100/1/p991.pdf

 

So to add to what Max said, you use site maps as a tactic in the hopes of getting more of your site's pages indexed.

 

Submitting a sitemap makes spider easy to crawl and index your website, moreover you can also find crawler activity on your sitemap in google webmaster tool.

 

An on-site sitemap is like a road map for Google, an XML sitemap is the GPS. Both are important. The XML sitemap also give you the ability to identify all of your URLs, the frequesncy that your pages are updated, and also assign a priority to the most important pages, ie: product pages or pages taht you are optimizing to get indexed.

 

yeah I agree, I never bother with the sitemap, seems pointless.

 

Agree. Google will faster indexing your site if you submit a sitemap. It's not a problem if you don't because sooner or later your page will be indexed. It's just about the time. sitemap give Google direct way to your page.

 

Google might be good at crawling a site, but its not perfect. Its not uncommon for BIG sites, such as forum with over 1 million post and threads to only have 20,000 pages indexed with google. So where are the other 980,000 pages at.

That is where the sitemap comes in. It helps google and the other search engines find the pages of your site that it might have missed.

For small sites, with less then 100,000 pages, you may not need a sitemap.

For medium sites, with around 500,000 pages, you probably need a sitemap.

For large sites with over 1 million pages, you diffidently need a sitemap.

So depending on the size of your site, that will help determine whether you "really" need a sitemap or not.

To give you an idea of how important it is. A couple of months ago I made soem changes to my forum and disabled the sitemap by accident. In 6 weeks google delisted 280,000 of my indexed pages. This was back in april and may of 2009. My site still has not recovered from that mistake. It went from 12% growth and 290,000 indexed pages, to 3% growth and 23,000 indexed pages. Here we are 2 months later, and I still have not recovered those delisted pages.

With sitemap - 290,000 indexed pages.
Without sitemap - 23,000 indexed pages.

 

Sitemaps may help get you indexed quicker.

 

Thanks for the useful examples. With SEO, we have to remember that whatever you know you don't know everything and the things you do know don't necessarily apply to all situations

 

So, I don't have any websites that have near as many pages as ~kev~, but I have a question. If you have a site that has over 50,000 pages and Google only accepts sitemaps with less than 50,000 URLs, how do you submit a sitemap with 100,000 URLs? I know you are supposed to create two sitemaps, but do you submit the second under a different webmaster account or different URL through webmaster tools?

 

The sitemap is broke into multiple parts, with each part containing less then 50,000 urls. My sitemaps are set to 45,000 URLs each. And each night the forum script rebuilds the sitemap - all 12, 13, 14 or 15 sections - and google grabs the new build.

So if your forum or site has 400,000 URLs, with 40,000 urls in each sitemap, the script would need to build 10 sitemaps.

 

Yes, this is actually how I feel as well. I really can't see it any other way.

 

So ~kev~, what software do you use to build these sitemaps? It must allow you to break the sitemaps into 40,000 url segments? I am very interested.

 

sitemap is good but its not mandatory thing

 

Sitemap is good because it can index your site faster. Let me ask this then, how can sitemap NOT benefit you with your site? How can it not help you? Can sitemapping harm your site in any way?

I just can't understand what the harm in sitemapping is. If you guys can explain this, do so.

 

Site Map is important Factor of SEO side. If in your website add new page and if want to crawl page, then you can used the site map and submit site map in google webmaster tool.

 

It helps the spider the index your pages more easily from what I heard from SEO experts.