Google crawlers/spiders ip's

Discussion in 'Google' started by imfusa, Aug 30, 2012.

0
  1. #1
    Where can i find the ip's of the crawlers/spiders that google uses to index/crawl your website for all the google tlds?
     
    imfusa, Aug 30, 2012 IP
  2. shreeonline

    shreeonline Peon

    Messages:
    53
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I don't know but I will appreciate your question and I want to know the answer of this question.
     
    shreeonline, Aug 30, 2012 IP
  3. MatthewWoodward.co.uk

    MatthewWoodward.co.uk Banned

    Messages:
    196
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #3
    They use Googlebot as the browser agent to pick them up that way
     
    MatthewWoodward.co.uk, Aug 30, 2012 IP
  4. imfusa

    imfusa Active Member

    Messages:
    694
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    70
    #4
    Hmm, so that means they don't have a static ip for this this or it;s been changed every time?
     
    imfusa, Aug 31, 2012 IP
  5. MatthewWoodward.co.uk

    MatthewWoodward.co.uk Banned

    Messages:
    196
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I'm not sure but if you just identify them by user agent you dont have to worry about dynamic IP's and the like
     
    MatthewWoodward.co.uk, Sep 2, 2012 IP
  6. 3dy.ro

    3dy.ro Member

    Messages:
    100
    Likes Received:
    3
    Best Answers:
    1
    Trophy Points:
    45
    #6
    This. But to answer OP's question: No, they don't have static IPs, although sometimes you might catch them with the same IP (not on the same site, though... I doubt they'll crawl the same one without changing it).
     
    3dy.ro, Sep 3, 2012 IP
    Jim4767 likes this.
  7. mamtak

    mamtak Member

    Messages:
    229
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    28
    #7
    I do not know if you can do that?
     
    mamtak, Sep 4, 2012 IP
  8. ironcladservers

    ironcladservers Well-Known Member

    Messages:
    389
    Likes Received:
    20
    Best Answers:
    4
    Trophy Points:
    115
    #8
    This list is going to change from time to time as Google updates their network, thats why going by IP may not be the best way to track it. Reverse DNS on a Google crawler will always show ".googlebot.com" in the hostname.

    I'm not sure how updated this list is but this may help your cause:
    http://www.iplists.com/google.txt
     
    ironcladservers, Sep 4, 2012 IP