Site Stopped Getting Crawled after Sitemap Added

I just added a sitemap to a site, one that's honestly very small at only around 150 files to index. I added the sitemap the day before yesterday, and then watched my logs. Previously, I would get on average 15-20 hits per day from Googlebot in its various forms. Yesterday, all crawler traffic stopped except for Googlebot's getting the sitemap.xml file, which it pulled.

I tried updating the files and updating the sitemap 6 times yesterday, and each time it only grabbed the sitemap, and grabbed no other files.

The only variance from this is one "HEAD" bot, which did a HEAD of the index file, but other than that, nothing.

Anyone have ideas?

 

What program did you use to create the sitemap?

Download and install GsiteCrawler, it works like a bot and will report back any crawl problems giving you a better look at your site. You can also use the program to create a sitemap.xml file.

 

I read something like this happening to many people on google groups

 

I have just noticed the same thing to one of my large sites in the last few days

I have a site with 4000 pages , PR5, gets thousands of hits a day from google.

Yet, Now that I have uploaded a xml sitemap about 3 weeks ago the bots stopped coming.

The sitemap all works, cause at google/sitemaps, it shows all my results, says status ok, and so fourth

I have 3980 pages indexed as well.

So yea....Kinda weird that the bots stoped

I also have 1,180 backlinks so I know that googleknows where I am , the site is also 2 years old....

I wonder why .....

 

When is the last date of the bot crawl? I am looking at a site that hasnt been crawled in 5 days and has a sitemap attached.

 

My last crawl was around the 27th of Sept

 

Im seeing two sites with the 29th as the last crawl and both sitemaps have been updated and submitted since then.

I wonder if there is a stall due to the Google update.

 

I just checked my sitemaps.

Last downloaded : 14 hours ago

Cache date :as retrieved on Oct 3, 2006 23:22:53 GMT.

So looks like it just refreshed itself....

 

After submitting your sitemap if Google sees that it has crawled all pages in the sitemap it will slow down and only make occasional crawls to see if you have added new content.

 

So would you say its better not to add a sitemap? Thats the conclusion I came too after reading many stories on google groups

 

Can you linik me to some of those google groups.

I am interested in reading as well

 

Here is one example but I would suspect its more of the sandbox working as the example is on a new site. There are many discussions regarding sites dropping from the index after attaching a sitemap, even on DP.

What I do suspect is that if there are any issues with your site, no matter how small, Google Sitemaps will pick it up and bring chaos to the site.

I wouldn’t attach the sitemap.xml until the site is clean as whistle and to be honest I am not seeing any benefits of attaching a Google sitemap unless the site has thousands of pages.

Sundaybrew – my sitemap is being downloaded within a couple of hours of submitting but the bot is still sitting at the bus stop, still no crawl since Sept and no update on the Stats. Content has been added daily; the sitemap has been updated and submitted.

 

I never do sitemaps since google bot is kind of doing good job by itself on all my sites .... I have bots visiting the same day or day after domain is registered and website is up and it gets indexed within a week.... so no sitemap here

 

i had the same problem too earlier, but i tried a html sitemap and now google is indexing my pages.

 

You mihgt find answer to your question if you pay attention to such a thing as update frequency that can vary in your sitemap. In the tool you are using to generate .xml sitemap you can adjust your site's update frequency as well as importace of separate pages WITHIN the site (0.1 - 1.0). These two moments will certainly influence hoe often your site is crawled. Check it out by yourself!