1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Has google stopped crawling your site read this

Discussion in 'Google Sitemaps' started by davediab, Nov 2, 2005.

  1. #1
    Google didnt crawl my site anymore and i dont know why but still it would crawl the index leave come back crawl the index when there are plenty of links on the index it could have went to
    try creating a sitemap and google will be in love
    http://johannesmueller.com/gs/ this proggy works well to create it :)
    SEMrush
     
    davediab, Nov 2, 2005 IP
    SEMrush
  2. softplus

    softplus Peon

    Messages:
    79
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Google Sitemaps aren't magic - but I sure love that program :D :D :D

    Remember, Google Sitemaps only adds your URLs to the Google crawler queue, but Google will use its own criteria to figure out which URLs it really wants to crawl - so don't think it's a magic bullet to get indexed, and don't think you can forget about getting links, because you can't, ever ;)

    The better your site is standing with Google (ie better inbound links), the better Google Sitemaps makes sure that your site is fully indexed. So first get the links, then add the Sitemaps, and hope for the best.
     
    softplus, Nov 3, 2005 IP
  3. seolion

    seolion Active Member

    Messages:
    1,495
    Likes Received:
    97
    Best Answers:
    0
    Trophy Points:
    90
    #3
    The same might be happening to my site http://www.best-personal-loan-rate.co.uk which has an xml sitemap. Still only my index page is crawled and not my inner pages....:mad:

    My xml sitemap is at http://www.best-personal-loan-rate.co.uk/sitemap.xml
    Any body can figure out any solutions??:confused:
     
    seolion, Nov 3, 2005 IP
  4. scottj

    scottj Peon

    Messages:
    168
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Google has definitely crawled my sites more since I started using sitemaps. I really think that the sitemaps can be quite beneficial and will definitely use them on new sites in the future.
     
    scottj, Nov 10, 2005 IP
  5. vbrocks

    vbrocks Active Member

    Messages:
    90
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    98
    #5
    greetings,
    i've looked at your xml file and it seems like it hasn't updated since 2005-10-26. So i think that's the reason for the googly guy to ignore it...
    All i can suggest is to insert more recent times in the xml file. If you have vb6 you can use the source code of a small program i wrote for one of my sites - all it does it updates the sitemap.xml file with the new timestamps, whenever you run it. i use task scheduler on my server for that purpose. Works quite well.
    Well, the prog is too simple to recurse subdirs, but my sites are tiny, so i didn't care much.
    Ah, also - i suggest using changefreq attribute with the 'hourly' value ;) .

    vb6 source code - http://www.vbrocks.net/downloads/dir2xmlSRC.zip
    binary - http://www.vbrocks.net/downloads/dir2xml.zip
    sample xml - http://www.hiddencoupons.com/sitemap.xml

    --
    vbrocks.us
     
    vbrocks, Nov 12, 2005 IP
  6. NubKnacker

    NubKnacker Peon

    Messages:
    20
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #6
    That's the absolute worst thing you can do. Most webmasters assume that just because the cnagefreq is set to "hourly" google will visit your site more often. That's far from the case.

    If you have a static page set the changefreq to "weekly". Google will crawl the page more often because it doesn't want to provide an old page in it's search so it will try and make sure that it crawls atleast once a week to handle changes to the page.

    Forums for this reason are a bad place for google. The content changes so often that google knows it will never be able to keep up. (It can't crawl your site 24/7.) So it delays crawling the site by as much as possible to make sure it gets as much content as possible at a later stage. (There's no real proof of this, atleast not that I have seen.)

    An RSS feed for your site helps as well since it's much easier for google to crawl through an RSS feed (without the pesky HTML tags ;)). Ever since I put a RSS feed on a clients forum google has been there almost for the entire day. The RSS feed exports every 15 mins with the last 100 topics/posts. Try it, it might give you better results.
     
    NubKnacker, Nov 14, 2005 IP
  7. vbrocks

    vbrocks Active Member

    Messages:
    90
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    98
    #7
    that's quite interesting suggestion - do not update it hourly, update it every 15 minutes instead.
    could be true for some busy sites, mine is a slow one.
    My point was, actually, not to put 'hourly' in the sitemap.xml just for fun. I think that the real site update should happen with comparable frequency.
    regards,

    --
    vbrocks
     
    vbrocks, Nov 16, 2005 IP
  8. minstrel

    minstrel Illustrious Member

    Messages:
    15,082
    Likes Received:
    1,243
    Best Answers:
    0
    Trophy Points:
    480
    #8
    Not only is there no proof FOR this belief - there's lots of proof AGAINST it.

    My forums certainly do get crawled daily, seven days a week, and usually more than once a day. Last week was not an atypical week: my psychology forum received 1852 hits from Googlebot in 7 days. That forum luanched March 25, 2004: except for perhaps the first few weeks, there has not been a day since then when Googlebot did not crawl the site - even when I added a MOD which inadvertently delivered error pages to Googlebot it still kept coming daily to index the error pages.
     
    minstrel, Nov 19, 2005 IP
  9. dasher

    dasher Peon

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Hi All

    New to your forum.

    I have added a google sitemap to our site. The Frequency is set to monthly, but the pages are seldom if ever changed. Does this present a problem?

    Des
     
    dasher, Nov 19, 2005 IP
  10. Jackson_Hu

    Jackson_Hu Peon

    Messages:
    228
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #10

    according to the information in my google sitemap account ,google download my sitemap everyday,I would assume that helps for my site been indexed
     
    Jackson_Hu, Nov 26, 2005 IP
  11. SiteExpress

    SiteExpress Well-Known Member

    Messages:
    1,355
    Likes Received:
    153
    Best Answers:
    0
    Trophy Points:
    155
    #11
    I use a Google sitemap.xml, and Google crawls it daily. Sometimes twice.
     
    SiteExpress, Nov 26, 2005 IP
  12. hit4biz

    hit4biz Peon

    Messages:
    20
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    If you do not have a program to build a google friendly sitemap you can go to

    www.auditmypc.com/free-sitemap-generator.asp

    Nothing to install from there. Just run their proggy. It's a javascript and works very well. It spiders your site and produces a finde sitemap.
    All you have to do then is to upload the new sitemap with a ftp program (like filezilla) and include a link to it on your startpage. That's it.
    Not the best automated way, but a good one if you do not change your pages very often. AND it is one of those great free tools :D
     
    hit4biz, Apr 7, 2006 IP
  13. AmmoX

    AmmoX Guest

    Messages:
    47
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
    Remember guys it's still a Beta product and also don't forget that Google is doing this to help normalize their database and query data. If they can get web sites to start providing standard reporting data that makes the bots and systems behind the scense work better and faster.

    Just look at how many times the robots que is cleared. There obviously still tweaking.:cool:
     
    AmmoX, Apr 9, 2006 IP
  14. softplus

    softplus Peon

    Messages:
    79
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #14
    Just a quick note for all those discussing the crawl-frequency settings:

    They're ignored by Google at the moment. They are only using them to collect statistics. Same goes for the priority setting.

    Personally, I'd remove that tag from the xml file :D
     
    softplus, Apr 11, 2006 IP
  15. AmmoX

    AmmoX Guest

    Messages:
    47
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #15
    I knew that the priority tag was placed for future use. But after the last month or so it looked liked the freq tag was getting a tune up. I don't have anything other than visual information for this but the frequency of downloads seemed to have gone up. Noticed after the last robtos.txt reload.

    Of course this is all from just paying attention to the data on the sitemaps site, nothing more scientific than that.:confused:
     
    AmmoX, Apr 16, 2006 IP