1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Sitemap generator for large sites

Discussion in 'Google Sitemaps' started by Ocean Storm, Dec 30, 2007.

  1. #1
    Hi,

    Just wondering what everyones thaughts are about current site map softwares.

    I need a site map generator that can easilly handle more than 200,000 URL's

    Currently I am using I architect but when i get to over 17,000 urls it starts to freeze up on me.
    SEMrush
    So wondering what anyone else can suggest that is good on the current market
     
    Ocean Storm, Dec 30, 2007 IP
    SEMrush
  2. websitetools

    websitetools Well-Known Member

    Messages:
    1,508
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #2
    Try newest version of A1 Sitemap Generator - 1.5.5

    At least I know many who scan websites of such sizes without problems. That said, you may still want to configuring options for best memory usage.
     
    websitetools, Dec 31, 2007 IP
  3. dairyman

    dairyman Notable Member

    Messages:
    7,225
    Likes Received:
    274
    Best Answers:
    0
    Trophy Points:
    205
    #3
    The free site map generating tools are capable of providing codes to only less than 1000 pages. You have to look for some paid versions and that to you have to design the site map page in such a way that it is free from clutter and look neat. It has to be arranged in the following manner like topic, followed by sub topics and still possible you can go in for third sub division. :)
     
    dairyman, Jan 1, 2008 IP
  4. Ocean Storm

    Ocean Storm Peon

    Messages:
    117
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Hi dairyman
    Thanks but i am using a paid sitemap maker by Iarchitect

    The only thing i can think of is i need to filter a string or folder or somthing that may be causing my problem but im not exactly sure what.

    The problem lies since i have created a huge amount of pages and content with a script called affilistore. http://www.affilistore.com
    Which uses mysql database to generate content from datafeeds in affiliate networks.

    the section of my site which im still building and in which i use the affilistore product on is http://www.online-shop.com.au/catalogues If anyone is able to review the site if it gives better understanding??? or be able to help me so that my problem is solved that would be great.
     
    Ocean Storm, Jan 1, 2008 IP
  5. websitetools

    websitetools Well-Known Member

    Messages:
    1,508
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #5
    If you only want to use Iarchitect? Have you tried contacting their support? That is always #1 thing I do when I encounter problems with a product. 200.000 is quite large website (with prior versions of A1SG it could have been a problem as well). Anyways, perhaps you could just filter off http://www.online-shop.com.au/catalogues when scanning? Or do you want those pages included in sitemap?
     
    websitetools, Jan 2, 2008 IP
  6. Ocean Storm

    Ocean Storm Peon

    Messages:
    117
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Have contacted the tec support via their forum but very slow to reply

    I did particulary want the http://www.online-shop.com.au/catalogues to be included in the site map as this section alone makes up probably 90% of my website.

    I think i have found one of the problems and that is that if some one clicks on visit store it gives off a redirect link such as **http://www.online-shop.com.au/catalogues/go.php?proddb=2&l=2056
    each link has its own unique code number but the common part of the urls are go.php?proddb=

    Obviously it is still picking up these links because it has the http://www.online-shop.com.au/catalogues/ part in the redirect link.

    should i put /go.php?proddb= in the filiter or just go.php? or ????
     
    Ocean Storm, Jan 2, 2008 IP
  7. kmap

    kmap Well-Known Member

    Messages:
    2,215
    Likes Received:
    29
    Best Answers:
    2
    Trophy Points:
    135
    #7
    I can generate sitemap for you

    $50

    50% advance
    10 days to complete



    Regards

    Alex
     
    kmap, Jan 6, 2008 IP
  8. gigmax

    gigmax Peon

    Messages:
    73
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    A1 Sitemap Genertor, you can test 30 day.
     
    gigmax, Jan 8, 2008 IP
  9. mobernard

    mobernard Active Member

    Messages:
    187
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #9
    Why not make use of Google Sitemap generator?
     
    mobernard, Jan 8, 2008 IP
  10. neds75

    neds75 Peon

    Messages:
    201
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Problem with google sitemap generator is because that is python script, and python is not supported on all websites.
     
    neds75, Jan 12, 2008 IP
  11. neasneap

    neasneap Peon

    Messages:
    26
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    are these really work well?
     
    neasneap, Jan 12, 2008 IP
  12. trichnosis

    trichnosis Prominent Member

    Messages:
    13,782
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #12
    i suggest you to try a1 sitemap generator or gsitecrawler for large web sites
     
    trichnosis, Jan 14, 2008 IP
  13. neasneap

    neasneap Peon

    Messages:
    26
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
    i hope it works well if it is not i have to do my web site's sitemap by hand.
     
    neasneap, Jan 14, 2008 IP
  14. onlinemoney

    onlinemoney Banned

    Messages:
    691
    Likes Received:
    63
    Best Answers:
    0
    Trophy Points:
    0
    #14
    i too have the same problem with my site having more then 20k pages... any expert opinions plz who have tried or used for their sites :(
     
    onlinemoney, Jan 15, 2008 IP
  15. websitetools

    websitetools Well-Known Member

    Messages:
    1,508
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #15
    20k pages (or even 200k) is usually no problem at all. Consider trying A1 Sitemap Generator (newest version 1.5.9). Should you contrary to what I expect encounter any problems, please contact me either through email or Digital Point PM :)
     
    websitetools, Jan 15, 2008 IP
  16. nicolat

    nicolat Peon

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #16
    try this one: xml-sitemaps (dot) com
    it does up to 500 free and unlimited for $19

    Here is the brief summary of its features:

    1. It generates any kind of sitemap you require: XML, Text, HTML site maps.
    2. It is developed in PHP languages and works with most web-server's configurations
    3. Fast and easy installation onto the same host that should be crawled
    4. Flexible configuration that allows you to set any sitemap parameters and crawler settings
    5. Support of LARGE websites, dividing the sitemap on the parts per 50,000 URLs each and Sitemap Index file according to the Google sitemap protocol.
    6. "robots.txt" exclusion protocol is supported
    7. GZip compression supported (optional)
    8. User-friendly progress indication for sitemap generation in manual mode
    9. Allows to setup the cron job to create sitemaps without a user interaction
    10. Informs (ping) Google automatically when sitemap generation is complete
    11. Website structure analysis feature represents the tree-like layout of your pages distribution within folders
    12. The script collects all generated sitemaps details and provides the log changes in it, including the number (and the lists) of added and removed pages.
    13. The broken links are detected by the application and reported on the special page, providing also the URLs pages that refer to these bad links.
     
    nicolat, Feb 3, 2008 IP
  17. HiSoC8Y

    HiSoC8Y Active Member

    Messages:
    136
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    55
    #17
    is $19.99 for unlimited installations, or just a single domain?
     
    HiSoC8Y, Mar 9, 2008 IP
  18. thefreead.net

    thefreead.net Peon

    Messages:
    30
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #18
    If its for unlimited then really great..
     
    thefreead.net, Mar 9, 2008 IP
  19. HiSoC8Y

    HiSoC8Y Active Member

    Messages:
    136
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    55
    #19
    i bought it, it doesnt seem to be limited to one site.

    which is good, unlimited :)
     
    HiSoC8Y, Mar 9, 2008 IP
  20. in2clearsky

    in2clearsky Peon

    Messages:
    121
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #20
    Try A1 sitemap generator. I tried myself on 100K+ without problem. Most of all they offer 30 days trial so you can try it before buy it.
     
    in2clearsky, Mar 9, 2008 IP