How long does it usually take for a 100,000+ page site to be indexed by Google?

Discussion in 'Search Engine Optimization' started by dspohn23, Jan 15, 2008.

  1. #1
    How long does it usually take for a 100,000+ page site to be indexed by Google?

    I figure that Google probably indexes sites faster, that have strong quality backlinks. Is this generally true?

    I started my site in mid-December and have been steadily gaining pages in Google every few days. I currently have 140,000 pages on my site and around 10,000 of them are shown in the google index(using site:). Additionally, it shows in google webmaster tools that Google has recently been crawling 4,000+ plus pages a day.

    In your experience, how long has it taken Google to index your whole site?
     
    dspohn23, Jan 15, 2008 IP
  2. forcer

    forcer Peon

    Messages:
    151
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #2
    It took me 5 months to index 100000 pages. I was blocking them using robots.txt to prevent duplicate content though. What I found was that more unique content matters than links. Getting 100000 of unique content is very hard. I was acquiring lots of links as well to get to that number of indexed pages, definitely without links that process would be much slower.

     
    forcer, Jan 16, 2008 IP
  3. sweetfunny

    sweetfunny Banned

    Messages:
    5,743
    Likes Received:
    467
    Best Answers:
    0
    Trophy Points:
    0
    #3
    It all depends on your backlinks, and more importantly your deeplinks. I often get sites in the 150-200k page range indexed in 14-25 days.

    It's really all about getting nice deeplinks to all your major subcategory pages, if you just get homepage links it's going to take a lot longer no matter how good they are.
     
    sweetfunny, Jan 16, 2008 IP
  4. budhanes

    budhanes Peon

    Messages:
    245
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I have a 250k+ page site I created. Its been about 3 months. I'll let you know when its done! :). I'm starting to think its going to take foooooorrrreeeevvvvvvvveeeeerrrrrr!!! BTW- It is weather related. "city,state weather" is my page formatting with almost every city in the US....
     
    budhanes, Jan 16, 2008 IP
  5. dspohn23

    dspohn23 Peon

    Messages:
    370
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #5
    budhanes - how many of your pages from that site are in google's index?
     
    dspohn23, Jan 16, 2008 IP
  6. joey112

    joey112 Guest

    Messages:
    2,250
    Likes Received:
    31
    Best Answers:
    0
    Trophy Points:
    0
    #6
    With so many pages, usually a week maximum. It will be indexed pretty quick mate. There is a lot of potential with a site that big aswell. You could make a lot of money.
     
    joey112, Jan 16, 2008 IP
  7. markz07

    markz07 Peon

    Messages:
    21
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    can you share us how you do it. ???
     
    markz07, Jan 16, 2008 IP
  8. joey112

    joey112 Guest

    Messages:
    2,250
    Likes Received:
    31
    Best Answers:
    0
    Trophy Points:
    0
    #8
    With a site that big, you must be targetting hundreds if not thousands of different keywords. Simply builkd backlinks, a lot of backlinks. And you will be making a real lot of money very soon.

    My sites have 30 pages usually. Yours have 100 000. With the exact same SEO. YOu will recieve more than 10 000 more traffic, well the laws of probability state that.

    :D
     
    joey112, Jan 16, 2008 IP
  9. budhanes

    budhanes Peon

    Messages:
    245
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #9
    At 4250 at the moment. I see an increase about every week or two. It fluctuates up and down by 50 daily? Not sure why, just as long as it increases stadily over time. I once did a site like this with around 40k pages. It took so long that I basically give up onj it and quit watching and hoping, after about a year, it had around 75% indexed. I'm not sure you can ever get all of it indexed, I dont know. Right now my sitemap has 48,800 links in it. I'd just be happy getting those ranked. As far as "making alot of money" joey112, I'm not too sure I'll make "that" kind of money on it. I will be using it to pick up some adwords and promote another site of mine. I have some sites in my "site farm" that are for later use once they get listed better in the serps. I can explain better in a PM if your really interested, but wont post publically about my ideas.
     
    budhanes, Jan 17, 2008 IP
  10. mrmrdan

    mrmrdan Peon

    Messages:
    194
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    How do you even get a site that big... How long it takes to index depends on SEO factors, different for different websites and for different niches.
     
    mrmrdan, Jan 17, 2008 IP
  11. budhanes

    budhanes Peon

    Messages:
    245
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #11
    Apache Mod Rewrite and PHP
     
    budhanes, Jan 17, 2008 IP
  12. take2

    take2 Peon

    Messages:
    42
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #12
    You can't trust the "site:" command in google to tell you an accurate number of pages in the index. I have a 500,000+ page site that google only states has 35,000 pages. But I get Google referrals on over 100,000 different pages each month, so the 35,000 number is wrong.

    Can you check your web server logs to see which pages Google has crawled? The pages won't get in the index if they haven;t been crawled.

    Other factors:

    1. The linked distance your inner pages are from your homepage and/or other externally linked to pages will play a factor in determining of Google crawls the deeper pages.
    - Good: 2 - 4
    - Poor: 5 - 6
    - Bad: 7 or more

    2. Are each page's title tag and description 100% unique across your site?

    3. Do you have non-parameterized URLs for each page?
    - Good: /San-Francisco-California-Hotels
    - Poor: /index.php?city=san+fancisco&state=california&type=hotels

    4. Is each page linked to on your site? If so, how many internal links per page?
    - Good: 3 or more internal links
    - Poor: 1 link
    - Bad: 0 links
     
    take2, Jul 6, 2009 IP