1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Best Sitemap Generator for Large Sites

Discussion in 'Google Sitemaps' started by Lpe04, Feb 22, 2009.

  1. #1
    Any suggestions for a Sitemap creator that will do a large site, preferably with minimal resources and something that Google will accept.

    Thanks.
    SEMrush
     
    Lpe04, Feb 22, 2009 IP
    SEMrush
  2. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I'm going to give GSiteCrawler a shot since the webpage was very nice :) I'll let you know how it goes ;)
     
    Lpe04, Feb 22, 2009 IP
  3. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Ok, trying GSiteCrawler but it's taking forever. Seems to be a good program though. (Many options, been around for awhile, etc.)
     
    Lpe04, Feb 23, 2009 IP
  4. websitetools

    websitetools Well-Known Member

    Messages:
    1,508
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #4
    If you have a large website, then it is going to take time :) I can't speak for GSiteCrawler, but here's a list of things you can try to speed up crawling with A1 Sitemap Generator. Some of the tips are general and apply to more sitemapper tools.
     
    websitetools, Feb 23, 2009 IP
  5. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #5
    How long is a long time? Like a whole day? And is this bad for a server? Thanks.
     
    Lpe04, Feb 23, 2009 IP
  6. websitetools

    websitetools Well-Known Member

    Messages:
    1,508
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #6
    Dpeneds on how large a website you have. You can always try to compare the speed / tinker with the options described in the article. It also discusses serverload issues. If you have a very large website, then you may want to e.g. schedule the scan at night (At least A1SG supports this. I don't know about GSC, but you could always try ask the author.)
     
    websitetools, Feb 24, 2009 IP
  7. jimbob

    jimbob Active Member

    Messages:
    165
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    58
    #7
    jimbob, Feb 24, 2009 IP
  8. WendyIMC

    WendyIMC Member

    Messages:
    46
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    41
    #8
    WebCEO produces well organized and visually attractive sitemaps for large websites.
     
    WendyIMC, Feb 24, 2009 IP
  9. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #9
    xml-sitemaps.org is good but maxes out at 500 pages. Thanks though.
     
    Lpe04, Feb 27, 2009 IP
  10. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #10
    I've found that it is best to break the site up into pieces, and work from there. Has helped me tremendously, and can be done over time.
     
    Lpe04, Feb 27, 2009 IP
  11. Fix Johnson

    Fix Johnson Well-Known Member

    Messages:
    238
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    103
    #11
    I have looked at the online site-map generators and was always concerned that they would add their own content as a "hidden fee", anyone ever check this out...I personally would not even know where to look :)
     
    Fix Johnson, Mar 1, 2009 IP
  12. catanich

    catanich Peon

    Messages:
    1,921
    Likes Received:
    40
    Best Answers:
    0
    Trophy Points:
    0
    #12
    WE have use gSiteCrawler for years and are happy with it. Check your settings and make sure you are indexing only the web pages, not jpg, gif, etc.

    Also, we have about 38,000 pages. We start it when we leave the office and it is done in the morning. Don't run it from home during the day. The band width will kill you.
     
    catanich, Mar 1, 2009 IP
  13. mortgageloan

    mortgageloan Peon

    Messages:
    339
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #13
    will try gsitecrawler..hve tried xml-sitemaps.org that is good ..
     
    mortgageloan, Mar 1, 2009 IP
  14. zaaylo

    zaaylo Well-Known Member

    Messages:
    1,417
    Likes Received:
    36
    Best Answers:
    0
    Trophy Points:
    170
    #14
    try auditmypc.com s sitemap generator
     
    zaaylo, Mar 1, 2009 IP
  15. tmbtech

    tmbtech Peon

    Messages:
    58
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #15
    I've used xml-sitemaps.org, i bought their paid version. its great for large sites and u can set a cron job so it'll update google automatically.
     
    tmbtech, Mar 2, 2009 IP
  16. LindaAdams

    LindaAdams Active Member

    Messages:
    284
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    60
    #16
    thanks a lot, this sitemap really helped
     
    LindaAdams, Mar 3, 2009 IP
  17. mycomputerguide

    mycomputerguide Peon

    Messages:
    71
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #17
    I would suggest phpSitemap, setup is 1 minute max.
     
    mycomputerguide, Mar 3, 2009 IP
  18. bobthegiant

    bobthegiant Peon

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #18
    Can you run a python script?

    The google sitemap generator is the fastest I have seen, generates an 18000 map without issue.

    https://www.google.com/webmasters/tools/docs/en/sitemap-generator.html

    Setup a cron job to run the script as often as your content updates and it will send the new one off to google without any interaction.
     
    bobthegiant, Mar 3, 2009 IP
  19. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #19
    No, I saw that but it seemed a little complex, may be worth it though if it is very fast. Thanks Bob.

    Lance
     
    Lpe04, Mar 3, 2009 IP
  20. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #20
    So I guess the question is now, what is the fastest sitemap generator for a site?

    Is it fast? Faster than GSiteCrawler? The only problem with GSiteCrawler is it seems to load each and every page, which may slow it down (at least according to stats). Thanks.
     
    Lpe04, Mar 3, 2009 IP