googlebot, how to tell how deep the crawl?

Discussion in 'Traffic Analysis' started by cgchris99, Feb 16, 2006.

0
  1. #1
    I have a php based site that emails me when a certain page gets hit by the google bot.

    Is there a way I can tell how deep the bot crawls?
    Is there any application for this or a log reader?

    I have been reviewing the pages indexed by google. We have seen quite a few changes since BD started. Now we have a bunch of supplemental results. When I look at them, a lot are cached from July of 2005.:mad:

    thanks for any advice.
     
    cgchris99, Feb 16, 2006 IP
  2. Smyrl

    Smyrl Tomato Republic Staff

    Messages:
    13,740
    Likes Received:
    1,702
    Best Answers:
    78
    Trophy Points:
    510
    #2
    Monitor you log files.

    Shannon
     
    Smyrl, Feb 16, 2006 IP
  3. jackburton2006

    jackburton2006 Peon

    Messages:
    5,296
    Likes Received:
    282
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Your server should have a record of all the bots that have spidered your site. You can trace Googlebot's progress that way.
     
    jackburton2006, Feb 16, 2006 IP
  4. Jocelyn

    Jocelyn Guest

    Messages:
    140
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Log analyzers will help you with that. It's nice to know that the ten hits were bots that did hit ten pages one time and not just the homepage 10 times...
     
    Jocelyn, Feb 16, 2006 IP
  5. just-4-teens

    just-4-teens Peon

    Messages:
    3,967
    Likes Received:
    168
    Best Answers:
    0
    Trophy Points:
    0
    #5
    do site:yoursite.com/folder and then try site:yoursite.com/folder/folder
    and see what results are brought back.
     
    just-4-teens, Feb 16, 2006 IP
  6. igotdreams

    igotdreams Peon

    Messages:
    23
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    also the new google sitemaps will give that type of information
     
    igotdreams, Feb 23, 2006 IP