G Site crawler is crawling indefenitly. Could any one help?

Discussion in 'Google Sitemaps' started by Personaltrainer, Apr 5, 2007.

  1. #1
    I am using gsite crawler to generate google sitemaps and is successful in doing so with my sites except one :confused:.When the crawler reaches some particular page it starts crawling it again and again and never stops. What could be the possible reason? Could any one help?
     
    Personaltrainer, Apr 5, 2007 IP
  2. websitetools

    websitetools Well-Known Member

    Messages:
    1,513
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #2
    Impossible to say without you giving us the url.

    However, one reason might be if you have a page that dynamicly creates a link pointing to itself, but with a different query string. e.g. if :

    page.php?datetime=whatever-new-datetime-1
    links to
    page.php?datetime=whatever-new-datetime-2
    links to
    page.php?datetime=whatever-new-datetime-3
     
    websitetools, Apr 6, 2007 IP
  3. Personaltrainer

    Personaltrainer Peon

    Messages:
    198
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #3
    The Url of the site is http://www.personaltrainers.org
     
    Personaltrainer, Apr 9, 2007 IP
  4. Personaltrainer

    Personaltrainer Peon

    Messages:
    198
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Thank you for the advice. The problem is fixed and I have run the crawler successfully.
     
    Personaltrainer, Apr 12, 2007 IP
  5. weppos

    weppos Well-Known Member

    Messages:
    99
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    125
    #5
    Which was the issue?
    I'm using GSite too and I would love to avoid the same bug. :)
     
    weppos, Apr 16, 2007 IP
  6. websitetools

    websitetools Well-Known Member

    Messages:
    1,513
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #6
    I PM'ed personaltrainer with a further analysis following the track of my advice in my eariler post.

    It is a not a trivial thing to "protect" a crawler against, I know from my programming of my sitemap generator program. There are some cases that can be discovered and avoided, but the rest will take... near indefinitely crawling :)
     
    websitetools, Apr 22, 2007 IP