Google Crawler faster than light

Discussion in 'Site & Server Administration' started by plmerlin, Sep 29, 2005.

  1. #1
    I realized that each time when I put a new page, Google crawls it immediately, I mean in the next 5 min.

    I tried 4 times and 4 times I saw in the log Googlebot on the page.
    I thought yeah, it's the adsense bot so I plugged a page with no adsense module... same thing in the 5min Google was looking at it.

    Is google spying me?
    How can google know that I added a page?
     
    plmerlin, Sep 29, 2005 IP
  2. LikeMike

    LikeMike Peon

    Messages:
    69
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #2
    OMFG, I've been trying to get google to read my site for 2 months and you get spidered in less than 5 minutes! How do I know if the bot has come to my site???
     
    LikeMike, Sep 29, 2005 IP
  3. JoeO

    JoeO Peon

    Messages:
    431
    Likes Received:
    29
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Is it the google crawler, or google mediabot (which will crawl every page with adsense on it right after the first time it is accessed)?...
     
    JoeO, Sep 29, 2005 IP
  4. plmerlin

    plmerlin Guest

    Messages:
    490
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    118
    #4
    It's what I thought at first but the bot is crawl-66-249-65-35.googlebot.com I don't think it's the adsense bot also the last page I added doesn't have any adsense module.
     
    plmerlin, Sep 29, 2005 IP
  5. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #5
    You can't tell from the IP address, just the user agent.
     
    digitalpoint, Sep 29, 2005 IP
  6. plmerlin

    plmerlin Guest

    Messages:
    490
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    118
    #6
    I know but the adsense bot is represented by something like mediabot-66-249-65-35.google.com also the last page did have any adsense on it.
     
    plmerlin, Sep 29, 2005 IP
  7. randymorin

    randymorin Peon

    Messages:
    387
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Questions

    Do you have Google toolbar?
    Do you have Google desktop?
    Do you have an RSS feed?
     
    randymorin, Sep 29, 2005 IP
  8. randymorin

    randymorin Peon

    Messages:
    387
    Likes Received:
    18
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Which site do you want spidered?
     
    randymorin, Sep 29, 2005 IP
  9. plmerlin

    plmerlin Guest

    Messages:
    490
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    118
    #9
    I've Google toolbar only, not the desktop, I have RSS Feed and Google sitemap.

    May be it coming from there... I did try again a page without adsense and 10 min later the guy was there.... really like if G was over my shoulder... weird :)
     
    plmerlin, Sep 29, 2005 IP