Best way for a deep crawl?

Discussion in 'Google' started by General Grant, Feb 5, 2005.

0
  1. #1
    I found a sitemap script, but the output file was over 32 MB's, so that's out.

    What is the best way to get thousands of pages indexed? Google just doesn't seem to want to crawl my entire site.
     
    General Grant, Feb 5, 2005 IP
  2. Mia

    Mia R.I.P. STEVE JOBS

    Messages:
    23,694
    Likes Received:
    1,167
    Best Answers:
    0
    Trophy Points:
    440
    #2
    The best way? What's your PR? The higher the PR, the deeper the crawl. That is pretty much the way I understand it.
     
    Mia, Feb 5, 2005 IP
  3. dchoe

    dchoe Active Member

    Messages:
    64
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    93
    #3
    and having new content
     
    dchoe, Feb 5, 2005 IP
  4. Estrange

    Estrange Peon

    Messages:
    124
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I think it's depending of your site structure.
    Our site has been deep crawled after 9 days of it's first published with a couple ibls and PR0.
    IMO shallow site link stucture (not so deep subfolders etc.), simple file extensions (do not use long query strings) and more relevant ibls from relevant sites will help.
     
    Estrange, Feb 5, 2005 IP
  5. General Grant

    General Grant Well-Known Member

    Messages:
    318
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    160
    #5
    One site is a pr4, I guess I have to add some xml feeds or something to keep the pages updated.
     
    General Grant, Feb 5, 2005 IP
  6. bloodwrath

    bloodwrath Active Member

    Messages:
    1,364
    Likes Received:
    49
    Best Answers:
    0
    Trophy Points:
    90
    #6
     
    bloodwrath, Feb 5, 2005 IP
  7. General Grant

    General Grant Well-Known Member

    Messages:
    318
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    160
    #7
    General Grant, Feb 5, 2005 IP