How to make a google sitemap for a 10k+ pages site ?

Discussion in 'Google Sitemaps' started by poseidon, Oct 18, 2006.

0
Page 1 of 2
  1. #1
    There are lot's of website which makes online sietmap but they are slow. I also tried to use SiteMapBuilder.Net software but it was slow and I had to close it after 7 hours.

    SO I need a good free software where I can set the limit of pages say like 5000 so that I can be assured that the software can finish the sitemap. :)

    Regards
     
    poseidon, Oct 18, 2006 IP
  2. hubel

    hubel Grunt

    Messages:
    382
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    168
    #2
    The speed of sitemap generators vary but the most important factor is the speed of your own site serving up the pages.

    That may be your problem
     
    hubel, Oct 19, 2006 IP
  3. websitetools

    websitetools Well-Known Member

    Messages:
    1,513
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #3
    I agree. There are some websites where I can easily crawl 10.000+ pages in an hour. However, if your website e.g. contains a huge forum / database backend... That may slow a website scan quite a lot.
     
    websitetools, Oct 19, 2006 IP
  4. poseidon

    poseidon Banned

    Messages:
    4,356
    Likes Received:
    246
    Best Answers:
    0
    Trophy Points:
    0
    #4
    yes, all the pages are in database :( well the problem is there is no software which can stop the sitemap. I mean when it had crawled 8k pages I tried to stop the software but it didn't so I had to closed it. *sigh*

    Any online sitemap generator which crawls link of the page only ?
     
    poseidon, Oct 19, 2006 IP
  5. websitetools

    websitetools Well-Known Member

    Messages:
    1,513
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #5
    I can perhaps understand "free" (even though its against my interests as I sell sitemap generation software, hehe), but why online????

    For what it is worth, my sitemap generator can be stopped by user. (However, it does require you to hit the "stop" button yourself, you can currently not set a fixed limit)

    In general, to partially solve your "fixed limit" wish, you can in most products control the crawler with various path filters.
     
    websitetools, Oct 20, 2006 IP
  6. poseidon

    poseidon Banned

    Messages:
    4,356
    Likes Received:
    246
    Best Answers:
    0
    Trophy Points:
    0
    #6
    nice software thomas but looks a bit ccompliated. Can you tell me how to make a sitemap ? there are just too many options :p
     
    poseidon, Oct 20, 2006 IP
  7. websitetools

    websitetools Well-Known Member

    Messages:
    1,513
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #7
    The options are there for those who need them, but yes, there are LOTS of options for just about everything.

    However, most people won't need (although they might benefit from it) to set more than one/two options.

    After you download and install
    A1 Sitemap Generator it opens in:

    "Scan website | Paths"
    Set the option: "Website directory root path".
    Hit the button "Start scan" ... Some time passes... Done!

    Select the tab "Create sitemap | Core options"
    Set the option: "Sitemap builder mode" to "Google XML sitemap"
    Hit the button "Build now" ... You now have a Google XML sitemap !



    I guess I will be writing a "your first sitemap" article soon :)
     
    websitetools, Oct 20, 2006 IP
  8. hhheng

    hhheng Banned

    Messages:
    2,633
    Likes Received:
    37
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Can you try my free online sitemap generator? Gurantee! It can make a standard sitemap file, which is acceptable by almost all major search engines, including google.

    If you don't like the online one, you can also download an offline to your computer at Free Softwares at my website.
     
    hhheng, Oct 22, 2006 IP
  9. poseidon

    poseidon Banned

    Messages:
    4,356
    Likes Received:
    246
    Best Answers:
    0
    Trophy Points:
    0
    #9
    can thsi be stopped in the middle ? :)
     
    poseidon, Oct 22, 2006 IP
  10. Eagle Eyes

    Eagle Eyes Well-Known Member

    Messages:
    413
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    105
    #10
    How do i submit it to the google site map? :confused:
     
    Eagle Eyes, Oct 22, 2006 IP
  11. == Morgana ==

    == Morgana == Banned

    Messages:
    31
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #11
    Check this out:

    http://www.dbnetsolutions.co.uk/gmapper
     
    == Morgana ==, Oct 22, 2006 IP
  12. == Morgana ==

    == Morgana == Banned

    Messages:
    31
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Sign in here http://www.google.com/webmasters/sitemaps with your gmail username/password
     
    == Morgana ==, Oct 22, 2006 IP
  13. trichnosis

    trichnosis Prominent Member

    Messages:
    13,785
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #13
    i have 10k pages for one of my sites. i have prepared a rss feed for every category:D (i have nearly 120 categories for this web site)
     
    trichnosis, Oct 25, 2006 IP
  14. dito199

    dito199 Peon

    Messages:
    60
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #14
    Try GSiteCrawler. You can pause and resume since it store the pages in db. You have the option to filter whats to ignore, and increase the speed by increasing the no. of crawlers.

    http://gsitecrawler.com

    But read my feedback on using the tool in this thread
     
    dito199, Oct 25, 2006 IP
  15. poseidon

    poseidon Banned

    Messages:
    4,356
    Likes Received:
    246
    Best Answers:
    0
    Trophy Points:
    0
    #15
    looks nice and I can do my work before 1st december. Ofcourse its good than I even wont mind paying for it. and thanks for the tips. Will surely set filters for images etc. Also what is the URL limit for google sitemap ? is it 25k links at max in sitemap?
     
    poseidon, Oct 25, 2006 IP
  16. dito199

    dito199 Peon

    Messages:
    60
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #16
    dito199, Oct 25, 2006 IP
  17. matrafox

    matrafox Active Member

    Messages:
    164
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    88
    #17
    If you have the link in database. Why don't you create sitemap yourself ? With a little php script ?
     
    matrafox, Oct 26, 2006 IP
  18. pctec

    pctec Well-Known Member

    Messages:
    1,623
    Likes Received:
    210
    Best Answers:
    0
    Trophy Points:
    160
    #18
    You can make a sitemap from your local files on your computer using CoffeeCup Google SiteMapper...
     
    pctec, Oct 27, 2006 IP
  19. websitetools

    websitetools Well-Known Member

    Messages:
    1,513
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #19
    The Google limit is 50k urls / 10MB per sitemap file.

    Most/many sitemap generators support:
    • splitting into multiple sitemap files + GZip.
    • scanning local file system
    • scanning localhost webserver
    • various path and file extension crawler filters

    Anyways, I kinda agree. If you already have all links in a DB, you should consider making a PHP script.
     
    websitetools, Oct 27, 2006 IP
  20. hoprofit

    hoprofit Peon

    Messages:
    117
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #20
    Why not modify the source code and generate sitemap "direct" from database?
    Please kindly allow me to share my experience. http://www.hoprofit.com/sitemaps/index.php. Note that all the latest post will be show in this page. :)
     
    hoprofit, Oct 27, 2006 IP
Page 1 of 2