1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How to create sitemap for large content site.

Discussion in 'Google Sitemaps' started by rahman15, Feb 26, 2006.

  1. #1
    I have article website which have 80,000+ articles. I was trying to create a sitemap with a free program called GSitecrawler; it’s been running 6 hours, still crawling the site.
    I wonder how long it should take to create the sitemap.
    Is there any better approach/tool to make sitemap for large site?
    SEMrush
     
    rahman15, Feb 26, 2006 IP
    SEMrush
  2. Momo

    Momo Peon

    Messages:
    965
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Momo, Feb 26, 2006 IP
  3. softplus

    softplus Peon

    Messages:
    79
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #3
    For really large sites I would prefer to use a sitemap generator that is tied in to the CMS / database. It just doesn't make sense for you to crawl the site when you (should) have all the URLs in your database, just a few queries away. Of course if your site is a combination of lots of small database-based components then you won't have a single table to query :D.

    Large sites take time to crawl, no matter which tool you use. For the GSiteCrawler you might want to try the SQL-based version (a slightly faster database makes a difference) or even use it in a network (you can have it run for the same site on several computers at the same time).
     
    softplus, Mar 14, 2006 IP
    Riboflavin likes this.
  4. Sarangan

    Sarangan Active Member

    Messages:
    127
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    58
    #4
    Just crawl them on directory based.. It is more easier..
     
    Sarangan, Mar 18, 2006 IP
  5. aprilforshee

    aprilforshee Peon

    Messages:
    47
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I might be too late with my response, but I use Sitemap Writer Proand it's crawling takes several minutes the most, plus, it has up to 40 multi-thread crawler.
     
    aprilforshee, Jun 4, 2010 IP
  6. aprilforshee

    aprilforshee Peon

    Messages:
    47
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Here is the website: http://sitemapwriter.com/
     
    aprilforshee, Jun 11, 2010 IP
  7. peterjoy

    peterjoy Peon

    Messages:
    19
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    For these there are so many facilities can be available for creating a large content map in a site. It is a combination of small data base components and give so many advance facilities for these content site map.
     
    peterjoy, Aug 12, 2010 IP
  8. iwillguide

    iwillguide Peon

    Messages:
    632
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #8
    If it is large content site with word press then it is easy to create.Pick the google XML site map plugin and activate it.
     
    iwillguide, Aug 13, 2010 IP
  9. aprilforshee

    aprilforshee Peon

    Messages:
    47
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    aprilforshee, Jan 27, 2011 IP
  10. kamaljit1987

    kamaljit1987 Member

    Messages:
    240
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #10
    kamaljit1987, Jan 28, 2011 IP
  11. lkraj

    lkraj Greenhorn

    Messages:
    72
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    18
    #11
    lkraj, Feb 8, 2011 IP
  12. websitetools

    websitetools Well-Known Member

    Messages:
    1,510
    Likes Received:
    25
    Best Answers:
    4
    Trophy Points:
    170
    #12
    You could also try A1 Sitemap Generator, but do note that 80k URLs always going to take time unless you are connected directly to DB. i.e. crawling takes time and often the webserver can choke and be the limiting factor, e.g. if you beef up simultaneous connections/threads in programs like A1SG

    (BTW, regarding the "limit of URLs in each sitemap" most sitemappers can split across multiple xml sitemap files)
     
    websitetools, Feb 8, 2011 IP