1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Best Sitemap Generator for Large Sites

Discussion in 'Google Sitemaps' started by Lpe04, Feb 22, 2009.

  1. mateon3000

    mateon3000 Peon

    Messages:
    31
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #21
    Lpe04: How many pages is the website?
    SEMrush
     
    mateon3000, Mar 3, 2009 IP
    SEMrush
  2. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #22
    Anywhere from 10,000 to 40,000 pages or more depending on how I spider it.'

    Thanks.
     
    Lpe04, Mar 3, 2009 IP
  3. mateon3000

    mateon3000 Peon

    Messages:
    31
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #23
    Ive heard good things about googles site map generator. Its free and also automatically generates new sitemaps for you when you add pages, change content etc.

    The pitfall is that it is still in beta, uses up resources on your server, might be buggy and more then likely the support for it might not be up to snuff.

    type in google

    google sitemap generator - Google Code
     
    mateon3000, Mar 3, 2009 IP
    Lpe04 likes this.
  4. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #24
    ok thanks I'll check it out. Hopefully it doesn't eat up too many resources.
     
    Lpe04, Mar 3, 2009 IP
  5. mateon3000

    mateon3000 Peon

    Messages:
    31
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #25
    Good luck. Post back letting us know the pros and cons of the beta sitemap generator.
     
    mateon3000, Mar 4, 2009 IP
  6. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #26
    Ok, if I ever get it working. I have noticed a couple of free download ones too. thanks. Best of luck to you too.
     
    Lpe04, Mar 4, 2009 IP
  7. onlinehowto.net

    onlinehowto.net Peon

    Messages:
    48
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #27
    Just today played again with that phyton generator. (it seams working fine since 2005 as there is no new release)
    It counts 12000 pages pretty fast and the autosubmit option is nice, though when I saw it for the first time it was looking too complicated.

    Install free python windows/Linux does not matter and set scheduler to run it as per your need. The whole setup wont take more than 1/2 hour if you follow the step-by-step directions.
    You will by pleased by it I am sure.
     
    onlinehowto.net, Mar 4, 2009 IP
  8. bobthegiant

    bobthegiant Peon

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #28
    IMO the google one is the fastest and most comprehensive. It also pings google to give the new sitemap which makes it very easy.

    If you can not figure it out you can change the settings in it to give you errors. There is two files the config file and the main python script.

    I just manipulated the config file and fixed the errors it gave out one by one.
     
    bobthegiant, Mar 5, 2009 IP
  9. websitetools

    websitetools Well-Known Member

    Messages:
    1,508
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #29
    Some sitemapper options (at least they are available in A1 Sitemap Generator) to try:

    * Have crawler stick to "GET" requests instead of also making "HEAD" requests
    (on a small percentage of websites/webservers this makes a huge difference)
    * Set up/down simultaneous connections/threads
    * Schedule sitemapper to crawl different time of day (i.e. when server load is low)

    Notice that some webservers automatically throttle... or worse completely choke when hit hard... So you may need to experiment a little to get max throughput.

    Another important thing is how "close" you are to the server...
    (Which give installed scripts an advantage, in particular if bandwidth is a problem)
     
    websitetools, Mar 5, 2009 IP
    Lpe04 likes this.
  10. mateon3000

    mateon3000 Peon

    Messages:
    31
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #30
    Hi bob,

    Whats your take on the google sitemap generator? Im thinking about implementing on my server but beta softwares always worry me when system is in production.
     
    mateon3000, Mar 5, 2009 IP
  11. bobthegiant

    bobthegiant Peon

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #31
    Its perfectly fine, I don't just use that one I use about 4 or 5 different generators as each gives different results.

    The google one though gives the most content and is alot more configurable. It is hardrer to setup than your average php script but once its going there is little to be done to it and it is alot more configurable than the others.

    Give it a go and see if you like it for yourself!
     
    bobthegiant, Mar 5, 2009 IP
  12. mateon3000

    mateon3000 Peon

    Messages:
    31
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #32
    Thanks bob,

    I'll try it out. Ive heard good things about it but havent talked to anyone that has actually been using it in production.
     
    mateon3000, Mar 6, 2009 IP
  13. linkgod

    linkgod Banned

    Messages:
    35
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #33
    If you use wp, you can using xml sitemap generator plugin
     
    linkgod, Mar 6, 2009 IP
  14. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #34
    Good info, thanks. I'm a little hesitant to purchase a full one at the moment. Might give Google's a try.
     
    Lpe04, Mar 7, 2009 IP
  15. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #35
    Only problem is, if you ever want to customize WP, it will break the plugin.
     
    Lpe04, Mar 7, 2009 IP
  16. ProTrac

    ProTrac Peon

    Messages:
    21
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #36
    I also have looked at the online site-map generators and found that they do add their own content in the xml file. But I edit that line out upload the file and everything is okay.. But for a really big site with thousands of pages I wouldn't recommend it, it's just to time consuming looking through all the urls for anything hidden.
     
    ProTrac, Mar 8, 2009 IP
  17. naguchennai

    naguchennai Active Member

    Messages:
    46
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    56
    #37
    Really there is no good sitemap generator in the net for LARGE SITE like 2 lacks and above urls... You can only try with any php script with database interface...

    that is true
     
    naguchennai, Mar 8, 2009 IP
  18. iseree

    iseree Peon

    Messages:
    129
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #38
    I use the paid version of xml-sitemaps.org. It runs very fast and google love it.
     
    iseree, Mar 8, 2009 IP
  19. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #39
    PHPWebsite seems to only search your server for folders and files based on the readme file I read, maybe not the best solution. Also Google's sitemap generator requires command line access to your server for install, maybe not the best solution for (or available) for some.

    Take care.
     
    Lpe04, Mar 8, 2009 IP
  20. kkobashi

    kkobashi Peon

    Messages:
    41
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #40
    I would just start off with GSiteCrawler at first and crawl the site in entirety until you get all the links fixed and everything looks solid. You will have a baseline sitemap to work with.

    THEN append as you go with a text editor by hand or program your CMS to append to the sitemap XML file after you submit and the page goes live. In a multi user editing environment you're going to want to program all that in.
     
    kkobashi, Mar 11, 2009 IP