Sitemap with over 1m entires?

Discussion in 'Google Sitemaps' started by Virtual Banker, Feb 21, 2009.

0
  1. #1
    I have a wordpress blog with over 1million posts.

    What is the best way to arrange my sitemap(s) for proper indexing?
     
    Virtual Banker, Feb 21, 2009 IP
  2. shailendra

    shailendra Peon

    Messages:
    1,225
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    0
    #2
    shailendra, Feb 22, 2009 IP
  3. jainmanoj

    jainmanoj Peon

    Messages:
    161
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Use Sitemap index files (to group multiple sitemap files)
     
    jainmanoj, Feb 23, 2009 IP
  4. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I'm having trouble getting a large site indexed too. It is taking forever to generate a sitemap.
     
    Lpe04, Feb 23, 2009 IP
  5. Greg Carnegie

    Greg Carnegie Peon

    Messages:
    385
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Single sitemap cannot contain more 53000 links and cannot be bigger then 10MB, however the good part is that Google accepts gz compressed sitemaps.
     
    Greg Carnegie, Feb 23, 2009 IP
  6. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #6
    I've pretty much given up on creating a sitemap for the time being. There are ways to link sitemaps together if it's above 50,000 links, Google talks about it somewhere I think, the only problem is getting it. For me, this may end up being a once a month process.
     
    Lpe04, Feb 23, 2009 IP
  7. Lpe04

    Lpe04 Peon

    Messages:
    579
    Likes Received:
    15
    Best Answers:
    0
    Trophy Points:
    0
    #7
    GSiteCrawler has an option to add more spiders at one time, just found that out today, maybe that will help.
     
    Lpe04, Feb 27, 2009 IP
  8. lauttehupe

    lauttehupe Well-Known Member

    Messages:
    238
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    105
    #8
    Forget normal Sitemap Generators.

    My Website has 4.2 Mio Pages.

    I got serversite tool (selfmade), but it takes like 7h to generarte 1,2 M Pages sitemap. I can san like 200 pages each second but it still takes long and my server cpu usage is high, so i can only run it in the night. I have 8 Core with 16 GB ram :(
     
    lauttehupe, Feb 28, 2009 IP
  9. catanich

    catanich Peon

    Messages:
    1,921
    Likes Received:
    40
    Best Answers:
    0
    Trophy Points:
    0
    #9
    gSiteCrawler will create the multiple files (xml) needed by Google. There could be 250 for your site though?
     
    catanich, Mar 1, 2009 IP
  10. suganindia

    suganindia Peon

    Messages:
    520
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #10
    So most of you recommend gsite crawler for large sites...
     
    suganindia, Mar 1, 2009 IP