Sitemaps vs Indexed pages

Discussion in 'Google Sitemaps' started by andymoo, Jan 15, 2010.

  1. #1
    I've got some pretty mental sitemap stuff going on, five million urls split into multiple maps, map index and automated weekly updates followed by automated notification of update to Google, Bing, Ask and Moreover.

    Who else would you ping / notify?

    Yes, it's a lot of URLs! We've got nearly five million legal downloads on the site so it's justified and not just a spam factory of gazillions of pages.

    When I google site:myurl it returns just over half a million pages most of the time.

    Whilst it's a huge figure it's nowhere near the number of pages I've submitted.

    The maps have been in place about three months now.

    How long in other folks experience has it taken Google to fully ingest and index the content of your sitemaps?

    Any tips?

    I know maps alone don't bring results and there are other factors but what can I do to make the indexing of my content faster?

    Or, am I being impatient and think it's going to take even a super computer quite to time to read five million pages?

    Please don't suggest an off site deep link campaign, it'd take 100 links x 5 million songs and that's beyond madness!

    :)

    Thanks
     
    andymoo, Jan 15, 2010 IP
  2. sharma

    sharma Active Member

    Messages:
    225
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    55
    #2
    site:myurl doesn't return all the pages indexed by google. Only way to check is in google.com/webmasters and under sitemaps. It will show how many urls are there in the sitemap and how many are indexed by google. You can set the custom crawl rate , increase it if your server can handle the overload but remember if google starts crawling your site more than other search engines will follow suite especially the chinese bots.
     
    sharma, Jan 17, 2010 IP
  3. andymoo

    andymoo Peon

    Messages:
    169
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Well GWMT says:

    Sitemap stats
    Total URLs: 5,255,283
    Indexed URLs: 170,505

    Site: = 503,000

    Too much variation there!

    As for chinese bots, they shouldn't be an issue as I've over 2,000 lines of rules in the firewall banning all Chinese traffic.

    And I'm not changing the crawl rate as it says not to unless the server is having issues and that's far from the case.

    Thanks
     
    andymoo, Jan 18, 2010 IP
  4. Jenny23

    Jenny23 Peon

    Messages:
    44
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I find that it is very hard to get all your pages indexed, even if you have a website with only a few hundred pages, does everyone agree on that?
     
    Jenny23, Jan 18, 2010 IP
  5. Lex350

    Lex350 Notable Member

    Messages:
    1,889
    Likes Received:
    31
    Best Answers:
    0
    Trophy Points:
    215
    #5
    Yes, Google specifically says they don't guarantee all your pages will be indexed. Get more backlinks and this will help.
     
    Lex350, Jan 18, 2010 IP
  6. Brandon Sheley

    Brandon Sheley Illustrious Member

    Messages:
    9,721
    Likes Received:
    612
    Best Answers:
    2
    Trophy Points:
    420
    #6
    get more backlinks
    get deep backlinks
    don't worry about your sitemap
    is your sites navagation setup correctly? can you access all 5 million pages with your breadcrumbs or some page on your site that links to the major sections?

    Sitemaps aren't the end all cure for indexed pages, if your site isn't indexed fully, it needs more backlinks or better page navagation
     
    Brandon Sheley, Jan 20, 2010 IP
  7. andymoo

    andymoo Peon

    Messages:
    169
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #7
    I said without suggesting deep links as that would totally take the p with the number of items I've got for sale, as for what you say about navigation, there's always room for improvement as there is on pretty much most sites.
     
    andymoo, Jan 20, 2010 IP
  8. Brandon Sheley

    Brandon Sheley Illustrious Member

    Messages:
    9,721
    Likes Received:
    612
    Best Answers:
    2
    Trophy Points:
    420
    #8
    links are your only answer
    you need more of them if you want the full site indexed
    it's not our fault you don't want to build a few links :rolleyes:

    ps.. you don't have to link to all 5million waraz pages, but the main sections would be good
     
    Brandon Sheley, Jan 20, 2010 IP
  9. andymoo

    andymoo Peon

    Messages:
    169
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Are you trying to take the piss there by calling my site warez?

    Cheeky tw@t
     
    andymoo, Jan 20, 2010 IP
  10. johagulo

    johagulo Peon

    Messages:
    879
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #10
    One tool that can assist with indexing is Google Webmaster Central
     
    johagulo, Jan 20, 2010 IP