Deep crawl

Discussion in 'Google' started by Lucky Bastard, Jan 24, 2005.

  1. #1
    If I had a site that was a 1 page landing page before (that page went up late last November). Last Google PR update that index page went up to a PR4 from a PR1 (thanks to some new backlinks).

    I have since got a new site up on the domain (the landing page was only a temp thing while I developed the main site) which Google comes to on a fairly regular basis (daily almost). However he is yet to crawl deeper in the site (which has now 2000+ pages.). I have had the new site up for less than month. How long does it usually take Google to start deep-crawling a site (pr4 if it makes any difference)?

    Any tricks for making Googlebot work harder and more adventurous? :) !!

    Go Google, Go!!
     
    Lucky Bastard, Jan 24, 2005 IP
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    The only thing you can do is get more links.
     
    digitalpoint, Jan 24, 2005 IP
  3. Lucky Bastard

    Lucky Bastard Peon

    Messages:
    406
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Thanks Shawn for your reply.

    I have gotten a heap more links since putting the new site up, plus the site has been getting links for the coop (few hundred now) again since putting the new site up. Also yeah I certainly am going to continue to get more links :) No time to sit pretty!
     
    Lucky Bastard, Jan 24, 2005 IP
  4. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #4
    All you can do is wait then.
     
    digitalpoint, Jan 24, 2005 IP
  5. dfsweb

    dfsweb Active Member

    Messages:
    1,587
    Likes Received:
    55
    Best Answers:
    0
    Trophy Points:
    88
    #5
    One of my work sites was a PR4 when I initially started working on it and Gbot used to do a deep draw every 4 weeks. I started getting more links and also started updating the pages after each deep crawl (small changes to the text) and Gbot started doing deep crawls more and more often ..... every 2 to 3 days.

    I don't update the site that often now and deep crawls have gone back to fortnightly. I didn't see any effect on SERPs in any case, and didn't want to lose bandwidth to unneccesary crawls.

    Anyway, all you need to do is update the home page (landing page). Make some BIG changes on this page, and GBOT might be tempted to do a deep crawl sooner than later! Good luck :)
     
    dfsweb, Jan 24, 2005 IP
  6. fryman

    fryman Kiss my rep

    Messages:
    9,604
    Likes Received:
    777
    Best Answers:
    0
    Trophy Points:
    370
    #6
    Best thing to do is put up a sitemap and link to it from your homepage. Show googlebot the way to get to your site.
     
    fryman, Jan 24, 2005 IP
  7. Bernard

    Bernard Well-Known Member

    Messages:
    1,608
    Likes Received:
    107
    Best Answers:
    0
    Trophy Points:
    185
    #7
    Are your 2000 plus pages developed using a CMS or dynamic structure? Perhaps there is an issue with session IDs, too many parameters, parameters being confused for session IDs, requiring cookies, your robots.txt file, etc. etc.

    There could be a number of reasons why spiders are not digging deeper...
     
    Bernard, Jan 24, 2005 IP
  8. wendydettmer

    wendydettmer Peon

    Messages:
    1,462
    Likes Received:
    70
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Sorta on topic question - does google penalize for putting up too many pages too quickly? Not that it's likely to happen for me, as i hand code in my spare time :) But i'm just more curious then anything...

    wendy
     
    wendydettmer, Jan 24, 2005 IP
  9. dfsweb

    dfsweb Active Member

    Messages:
    1,587
    Likes Received:
    55
    Best Answers:
    0
    Trophy Points:
    88
    #9
    I don't really have any data to back this up, but I would seriously doubt it. A lot of people would set-up a local copy and then upload the changes once tested. If somebody makes a BIG change to their website, they would have a lot of new pages on the website. I doubt if this will be penalised by Google, but would love to hear from anyone who thinks otherwise.
     
    dfsweb, Jan 24, 2005 IP
  10. Bernard

    Bernard Well-Known Member

    Messages:
    1,608
    Likes Received:
    107
    Best Answers:
    0
    Trophy Points:
    185
    #10
    I'm not aware of any penalty per se, but putting up a bunch of new pages will affect the PR distribution of the existing pages (likely diluting it) when factoring in the new pages (and site structure).
     
    Bernard, Jan 24, 2005 IP
  11. wendydettmer

    wendydettmer Peon

    Messages:
    1,462
    Likes Received:
    70
    Best Answers:
    0
    Trophy Points:
    0
    #11
    ahhh
    well, my site (30 lil pages that it is lol) still has a PR of 0, so i'm not worried :)

    thanks for the replies!
     
    wendydettmer, Jan 24, 2005 IP
  12. dfsweb

    dfsweb Active Member

    Messages:
    1,587
    Likes Received:
    55
    Best Answers:
    0
    Trophy Points:
    88
    #12
    I would have to disagree with that. Every page that's added to a site increases it's overall PR. Moreover, PR updates don't (IMO) take place every day. Google loves content!! So, if you are looking at adding page, do it. The more, the merrier! :D
     
    dfsweb, Jan 24, 2005 IP
  13. Bernard

    Bernard Well-Known Member

    Messages:
    1,608
    Likes Received:
    107
    Best Answers:
    0
    Trophy Points:
    185
    #13
    dfsweb, if you have a site with 10 pages and each page has a PR 4, then you add 50,000 new pages (say, one for each zip code or something), depending upon the link structure, the actual PR (not toolbar PR) for the existing 10 pages is going to be distributed across the 50,000 new pages. The 50,000 new pages will indeed add an actual PR value of 1, but this is not a toolbar PR of 1. Your original 10 pages' PR will drop unless you acquire enough new IBLs to offset the redistribution of PR to the new pages.

    The amount of PR redistribution will depend upon how heavily linked the existing pages are to the new pages.
     
    Bernard, Jan 25, 2005 IP
  14. dfsweb

    dfsweb Active Member

    Messages:
    1,587
    Likes Received:
    55
    Best Answers:
    0
    Trophy Points:
    88
    #14
    Bernard, the PR of a page depends on incoming BLs and not outgoing BLs from that page. Period! So, a PR5 page will remain a PR5 regardless of the number of outbound links on it.
     
    dfsweb, Jan 25, 2005 IP
  15. wendydettmer

    wendydettmer Peon

    Messages:
    1,462
    Likes Received:
    70
    Best Answers:
    0
    Trophy Points:
    0
    #15
    how often does google update their PR ratings?
     
    wendydettmer, Jan 25, 2005 IP
  16. Bernard

    Bernard Well-Known Member

    Messages:
    1,608
    Likes Received:
    107
    Best Answers:
    0
    Trophy Points:
    185
    #16
    dfsweb, as I understand it, the PR calculations are an iterative algorithm. The PR from IBLs are passed on with a dampening factor to the OBLs from a given page. When pages have a reciprical (even if chained and not direct) link structure, a portion of the PR given to the OBL targets will get passed back the original page through each successive iteration of the algo.

    People get all fired up arguing about PR leak. It's really a bunch of hot air over semantics. PR 'leak' is a misnomer, but commonly applied to explain the loss of potential circular PR redistribution through successive iterations of the calculation.

    In the scenario being discussed, the PR passed from the original 10 pages to the new pages will almost certainly diminish the circular redistribution more than the miniscule PR the new pages will pass back on the merit of just being a new page.
     
    Bernard, Jan 25, 2005 IP
  17. dfsweb

    dfsweb Active Member

    Messages:
    1,587
    Likes Received:
    55
    Best Answers:
    0
    Trophy Points:
    88
    #17
    The toolbar PR (which doesn't really matter) is updated quarterly. IMO, the real PR is also updated once in 2-3 months (at a different time to the toolbar update) and is complimented by big movements in SERPs as opposed to the smaller movements that can be accomplished by making changes to the content, title tags, back links etc.

    But, there is some speculation (Actually, a lot of speculation) that PR is updated on a regular basis (although I don't agree with it)
     
    dfsweb, Jan 25, 2005 IP
  18. dfsweb

    dfsweb Active Member

    Messages:
    1,587
    Likes Received:
    55
    Best Answers:
    0
    Trophy Points:
    88
    #18
    If your theory is correct, then the PR passed back from the new pages should not be miniscule due to the successive iterations. :D
     
    dfsweb, Jan 25, 2005 IP
  19. larysmith711

    larysmith711 Notable Member

    Messages:
    3,374
    Likes Received:
    341
    Best Answers:
    0
    Trophy Points:
    215
    #19
    I believe it's updated on a regular basis. I launched this site on last Thursday only about six days old. The hoempage has been crawled everyday. Check it out: http://www.google.com/search?hl=en&lr=&ie=ISO-8859-1&c2coff=1&q=site:www.seo-kit.com+seo-kit

    Have you seen homepages of sites with low PR's get crawled everyday? I HAVEN'T.

    It's update continuously. :D
     
    larysmith711, Jan 25, 2005 IP
  20. wendydettmer

    wendydettmer Peon

    Messages:
    1,462
    Likes Received:
    70
    Best Answers:
    0
    Trophy Points:
    0
    #20
    Thank you :) I also know that PR isn't one of the bigger things for me to worry about, and i have a longggggggg way to go before i'll even get a PR 1 lol.

    thanks for humoring my questions :)

    wendy
     
    wendydettmer, Jan 25, 2005 IP