GSiteCrawler is too slow !

Discussion in 'Google Sitemaps' started by Tim Smith, Jul 17, 2008.

0
  1. #1
    Dear all,
    I am using GSiteCrawler to craw my site and generate sitemap.xml. There are about 40,000 urls but it takes me more than 1 night (around 16 hours) to finish crawling. Is it normal ?

    Do you suggest any other way to generate sitemap.xml faster ? Thank you.

    My Internet speed is 7M / s.

    Tim
     
    Tim Smith, Jul 17, 2008 IP
  2. lukazs

    lukazs Well-Known Member

    Messages:
    583
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    125
    #2
    Yes it's normal. You could lower pause between crawls in GSiteCrawler to 1. (default is 4seconds).
     
    lukazs, Jul 17, 2008 IP