1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Site Stopped Getting Crawled after Sitemap Added

Discussion in 'Google Sitemaps' started by JetteroHeller, Oct 2, 2006.

  1. #1
    I just added a sitemap to a site, one that's honestly very small at only around 150 files to index. I added the sitemap the day before yesterday, and then watched my logs. Previously, I would get on average 15-20 hits per day from Googlebot in its various forms. Yesterday, all crawler traffic stopped except for Googlebot's getting the sitemap.xml file, which it pulled.
    SEMrush
    I tried updating the files and updating the sitemap 6 times yesterday, and each time it only grabbed the sitemap, and grabbed no other files.

    The only variance from this is one "HEAD" bot, which did a HEAD of the index file, but other than that, nothing.

    Anyone have ideas?
     
    JetteroHeller, Oct 2, 2006 IP
    SEMrush
  2. cormac

    cormac Peon

    Messages:
    3,662
    Likes Received:
    222
    Best Answers:
    0
    Trophy Points:
    0
    #2
    What program did you use to create the sitemap?

    Download and install GsiteCrawler, it works like a bot and will report back any crawl problems giving you a better look at your site. You can also use the program to create a sitemap.xml file.
     
    cormac, Oct 2, 2006 IP
  3. grs123

    grs123 Member

    Messages:
    86
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    43
    #3
    I read something like this happening to many people on google groups
     
    grs123, Oct 3, 2006 IP
  4. sundaybrew

    sundaybrew Numerati

    Messages:
    7,296
    Likes Received:
    1,260
    Best Answers:
    0
    Trophy Points:
    560
    #4
    I have just noticed the same thing to one of my large sites in the last few days

    I have a site with 4000 pages , PR5, gets thousands of hits a day from google.

    Yet, Now that I have uploaded a xml sitemap about 3 weeks ago the bots stopped coming.

    The sitemap all works, cause at google/sitemaps, it shows all my results, says status ok, and so fourth

    I have 3980 pages indexed as well.

    So yea....Kinda weird that the bots stoped

    I also have 1,180 backlinks so I know that googleknows where I am , the site is also 2 years old....

    I wonder why .....
     
    sundaybrew, Oct 5, 2006 IP
  5. cormac

    cormac Peon

    Messages:
    3,662
    Likes Received:
    222
    Best Answers:
    0
    Trophy Points:
    0
    #5
    When is the last date of the bot crawl? I am looking at a site that hasnt been crawled in 5 days and has a sitemap attached.
     
    cormac, Oct 5, 2006 IP
  6. sundaybrew

    sundaybrew Numerati

    Messages:
    7,296
    Likes Received:
    1,260
    Best Answers:
    0
    Trophy Points:
    560
    #6
    My last crawl was around the 27th of Sept
     
    sundaybrew, Oct 5, 2006 IP
  7. cormac

    cormac Peon

    Messages:
    3,662
    Likes Received:
    222
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Im seeing two sites with the 29th as the last crawl and both sitemaps have been updated and submitted since then.

    I wonder if there is a stall due to the Google update.
     
    cormac, Oct 5, 2006 IP
  8. sundaybrew

    sundaybrew Numerati

    Messages:
    7,296
    Likes Received:
    1,260
    Best Answers:
    0
    Trophy Points:
    560
    #8
    I just checked my sitemaps.

    Last downloaded : 14 hours ago

    Cache date :as retrieved on Oct 3, 2006 23:22:53 GMT.

    So looks like it just refreshed itself....
     
    sundaybrew, Oct 5, 2006 IP
  9. minnseoelite

    minnseoelite Peon

    Messages:
    845
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #9
    After submitting your sitemap if Google sees that it has crawled all pages in the sitemap it will slow down and only make occasional crawls to see if you have added new content.
     
    minnseoelite, Oct 5, 2006 IP
  10. grs123

    grs123 Member

    Messages:
    86
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    43
    #10
    So would you say its better not to add a sitemap? Thats the conclusion I came too after reading many stories on google groups
     
    grs123, Oct 5, 2006 IP
  11. sundaybrew

    sundaybrew Numerati

    Messages:
    7,296
    Likes Received:
    1,260
    Best Answers:
    0
    Trophy Points:
    560
    #11
    Can you linik me to some of those google groups.

    I am interested in reading as well
     
    sundaybrew, Oct 5, 2006 IP
  12. cormac

    cormac Peon

    Messages:
    3,662
    Likes Received:
    222
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Here is one example but I would suspect its more of the sandbox working as the example is on a new site. There are many discussions regarding sites dropping from the index after attaching a sitemap, even on DP.

    What I do suspect is that if there are any issues with your site, no matter how small, Google Sitemaps will pick it up and bring chaos to the site.

    I wouldn’t attach the sitemap.xml until the site is clean as whistle and to be honest I am not seeing any benefits of attaching a Google sitemap unless the site has thousands of pages.

    Sundaybrew – my sitemap is being downloaded within a couple of hours of submitting but the bot is still sitting at the bus stop, still no crawl since Sept and no update on the Stats. Content has been added daily; the sitemap has been updated and submitted.
     
    cormac, Oct 6, 2006 IP
  13. iatbm

    iatbm Prominent Member

    Messages:
    5,157
    Likes Received:
    353
    Best Answers:
    0
    Trophy Points:
    360
    #13
    I never do sitemaps since google bot is kind of doing good job by itself on all my sites .... I have bots visiting the same day or day after domain is registered and website is up and it gets indexed within a week.... so no sitemap here
     
    iatbm, Oct 6, 2006 IP
  14. niceboy

    niceboy Well-Known Member

    Messages:
    1,170
    Likes Received:
    17
    Best Answers:
    0
    Trophy Points:
    128
    #14
    i had the same problem too earlier, but i tried a html sitemap and now google is indexing my pages.
     
    niceboy, Oct 12, 2006 IP
  15. Tugeza

    Tugeza Well-Known Member

    Messages:
    204
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    128
    #15
    You mihgt find answer to your question if you pay attention to such a thing as update frequency that can vary in your sitemap. In the tool you are using to generate .xml sitemap you can adjust your site's update frequency as well as importace of separate pages WITHIN the site (0.1 - 1.0). These two moments will certainly influence hoe often your site is crawled. Check it out by yourself!
     
    Tugeza, Oct 12, 2006 IP