Google (and other spider) IP addresses

Discussion in 'Site & Server Administration' started by BigChase, Oct 30, 2005.

  1. #1
    How to get a list of the IP addresses from which spiders crawl websites in order to filer them out of access log analysis?

    In other words, I want to disregard any traffic to our websites that are being generated by spiders so as to have a more accurate count of "human" traffic. To do so, need to know which IP numbers in the access_log are from Google's (and other search engines') spiders. Do such lists exist?

    Thank you..
     
    BigChase, Oct 30, 2005 IP
  2. MentaLRZ

    MentaLRZ Well-Known Member

    Messages:
    364
    Likes Received:
    19
    Best Answers:
    0
    Trophy Points:
    155
    #2
    Ur in luck, heres my up to date list from vb, use it as a guide :)
     

    Attached Files:

    MentaLRZ, Oct 30, 2005 IP
    frankm likes this.
  3. rehash

    rehash Well-Known Member

    Messages:
    1,502
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    150
    #3
    just look for those who requested robots.txt
     
    rehash, Nov 6, 2005 IP