How can i tell if google is crawling all my pages

Discussion in 'Google' started by McDoug, Nov 20, 2009.

  1. #1
    For the last year or so I have not been able to figure out if google is crawling all my pages. I really dont think it is because it is sort of a strange site and users add most of the content. How can I check?
     
    McDoug, Nov 20, 2009 IP
  2. SearchBliss

    SearchBliss Well-Known Member

    Messages:
    1,899
    Likes Received:
    70
    Best Answers:
    2
    Trophy Points:
    195
    Digital Goods:
    1
    #2
    Join Google Webmaster Tools and submit a sitemap of all of your pages. Then wait a day or two, and you will see how many pages were in the sitemap compared to how many Google indexed.
     
    SearchBliss, Nov 20, 2009 IP
  3. TheMonkII

    TheMonkII Active Member

    Messages:
    148
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    55
    #3
    You can always view your logs. Most stat programs keep track of when the bot last visited.
     
    TheMonkII, Nov 20, 2009 IP
  4. Montreal Classifieds

    Montreal Classifieds Active Member

    Messages:
    808
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    60
    #4
    Use site:yourdomain
    Click on "show options". You can see what pages are indexed and when.
    On Google Webmaster Tools you can see stats for google crawler and last visit date. You can also extract that info from your webserver logs.
     
    Montreal Classifieds, Nov 20, 2009 IP
  5. touchjijo

    touchjijo Active Member

    Messages:
    538
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    80
    #5
    Goto Google webmaster tools and take your sitemaps page. There you can see number of pages and number of pages indexed.
    orelse try searching for 'site:yourdomain.com' in google and you'll see all your Google page index.
     
    touchjijo, Nov 20, 2009 IP
  6. jason830

    jason830 Peon

    Messages:
    373
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Use google webmaster, it is a good analytics tool for you.
     
    jason830, Nov 20, 2009 IP
  7. webcosmo

    webcosmo Notable Member

    Messages:
    5,840
    Likes Received:
    153
    Best Answers:
    2
    Trophy Points:
    255
    #7
    Check the user agent on your log file.
     
    webcosmo, Nov 20, 2009 IP
  8. Traffic-Bug

    Traffic-Bug Active Member

    Messages:
    1,866
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    80
    #8
    This information is available for viewing in the google webmaster tools section. Further, if you submit a sitemap you can easily compare how many pages in sitemap vs how many pages actually indexed (discounting for duplicate content etc.)
     
    Traffic-Bug, Nov 20, 2009 IP
  9. gameutopia

    gameutopia Peon

    Messages:
    975
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #9
    I have a php script installed that logs google activity to a file. I can then view it online. It shows exactly which page google crawled, the time, and date. Kind of interesting and works well, just not easy to install for everyone. It can be used with most any php site on paid hosting where you can change file permissions and edit files. Some days I see google crawls many pages, other days not so many. Server logs and stat programs work, but they are hard to pick through and find only google when there are a bunch of others mixed in the server and stats logs.
     
    gameutopia, Nov 20, 2009 IP
  10. believer1984

    believer1984 Peon

    Messages:
    1,544
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #10
    My advice would be to get the google webmaster tool and verify your site so you see everything.

    :eek:
     
    believer1984, Nov 22, 2009 IP