My latest Google, MSN, Yahoo Spider IP list

Discussion in 'Search Engine Optimization' started by hdogan, Apr 18, 2008.

0
  1. #1
    IP blocks are CIDR formatted

    Google

    8.6.48.0/21
    64.68.80.0/21
    64.68.88.0/21
    64.233.160.0/19
    66.249.64.0/19
    72.14.192.0/18
    209.85.128.0/17
    216.200.251.112/28
    216.239.32.0/19

    MSN

    64.4.0.0/18
    65.52.0.0/14
    131.107.0.0/16
    202.96.51.128/25
    207.46.0.0/16
    207.68.128.0/18
    207.68.192.0/20

    Yahoo

    66.196.64.0/18
    66.228.160.0/19
    67.195.0.0/16
    68.142.192.0/18
    68.180.128.0/17
    72.30.0.0/16
    74.6.0.0/16
    202.160.176.0/20

    Please let me know if I forgot to add some IP block.

    Thanks.

    PS: Please move the thread if I post it on wrong forum.
     
    hdogan, Apr 18, 2008 IP
  2. hdogan

    hdogan Peon

    Messages:
    316
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I've checked http://www.iplists.com site and updated my list.
    And I've found that some of IP blocks are not registered by the company (ie, Google, Yahoo, MSN). Especially Korean and Chinese IP ranges. So I didn't include them.
    Here is the updated list in PHP array format and PHP function to check: 

    <?php 
$_SPIDER = array(); 
# Google 
$_SPIDER[] = "8.6.48.0/21"; 
$_SPIDER[] = "64.68.80.0/21"; 
$_SPIDER[] = "64.68.88.0/21"; 
$_SPIDER[] = "64.233.160.0/19"; 
$_SPIDER[] = "66.249.64.0/19"; 
$_SPIDER[] = "72.14.192.0/18"; 
$_SPIDER[] = "209.85.128.0/17"; 
$_SPIDER[] = "216.33.229.160/29"; 
$_SPIDER[] = "216.200.251.112/28"; 
$_SPIDER[] = "216.239.32.0/19"; 

# Yahoo 
$_SPIDER[] = "63.163.102.0/24"; 
$_SPIDER[] = "66.163.160.0/19"; 
$_SPIDER[] = "66.94.224.0/19"; 
$_SPIDER[] = "66.196.64.0/18"; 
$_SPIDER[] = "66.218.64.0/19"; 
$_SPIDER[] = "66.228.160.0/19"; 
$_SPIDER[] = "67.195.0.0/16"; 
$_SPIDER[] = "68.142.192.0/18"; 
$_SPIDER[] = "68.180.128.0/17"; 
$_SPIDER[] = "69.147.64.0/18"; 
$_SPIDER[] = "72.30.0.0/16"; 
$_SPIDER[] = "74.6.0.0/16"; 
$_SPIDER[] = "202.46.19.0/24"; 
$_SPIDER[] = "202.160.176.0/20"; 
$_SPIDER[] = "202.165.96.0/19"; 
$_SPIDER[] = "203.141.52.0/26"; 
$_SPIDER[] = "206.190.32.0/19"; 
$_SPIDER[] = "207.126.224.0/20"; 
$_SPIDER[] = "209.73.160.0/19"; 
$_SPIDER[] = "209.131.32.0/19"; 
$_SPIDER[] = "209.191.64.0/18"; 
$_SPIDER[] = "211.14.8.0/24"; 
$_SPIDER[] = "216.109.112.0/20"; 
$_SPIDER[] = "216.136.232.0/21"; 
$_SPIDER[] = "216.145.48.0/20"; 
$_SPIDER[] = "216.155.192.0/20"; 

# MSN 
$_SPIDER[] = "63.194.155.144/29"; 
$_SPIDER[] = "64.4.0.0/18"; 
$_SPIDER[] = "65.52.0.0/14"; 
$_SPIDER[] = "131.107.0.0/16"; 
$_SPIDER[] = "202.96.51.128/25"; 
$_SPIDER[] = "204.95.96.0/19"; 
$_SPIDER[] = "207.46.0.0/16"; 
$_SPIDER[] = "207.68.128.0/18"; 
$_SPIDER[] = "207.68.192.0/20"; 
$_SPIDER[] = "213.199.128.0/20";

# found this little function at php.net manual pages. 
function net_match($cidr, $ip) { 
  list($net, $mask) = explode('/', $cidr); 
  return (ip2long($ip) & ~((1 << (32 - $mask)) - 1)) == ip2long($net); 
} 

function is_spider() { 
  global $_SPIDER; 

  $count = count($_SPIDER); 

  for ($i = 0; $i < $count; $i++) { 
    if (net_match($_SPIDER[$i], $_SERVER['REMOTE_ADDR'])) { 
      return true; 
    } 
  } 

  return false; 
} 

# Example code 
if (is_spider()) { 
  echo "Hello spider :)\n"; 
} 
else { 
  echo "Umm, you're not a spider...\n"; 
} 
?>
    PHP:
     
    hdogan, Apr 18, 2008 IP