There have been two spiders all over my site the last few days. XGet and Golem. I'd never heard of either so did a whois on the IPs. Golem: 66.249.71.32 OrgName: Google Inc. OrgID: GOGL Address: 2400 E. Bayshore Parkway City: Mountain View StateProv: CA PostalCode: 94043 Country: US NetRange: 66.249.64.0 - 66.249.95.255 CIDR: 66.249.64.0/19 NetName: GOOGLE NetHandle: NET-66-249-64-0-1 Parent: NET-66-0-0-0-0 NetType: Direct Allocation NameServer: NS1.GOOGLE.COM NameServer: NS2.GOOGLE.COM Comment: RegDate: 2004-03-05 Updated: 2004-11-10 OrgTechHandle: ZG39-ARIN OrgTechName: Google Inc. OrgTechPhone: +1-650-318-0200 OrgTechEmail: ************@google.com XGet: 66.196.91.96 OrgName: Inktomi Corporation OrgID: INKT Address: 4100 East Third Avenue City: Foster City StateProv: CA PostalCode: 94404 Country: US NetRange: 66.196.64.0 - 66.196.127.255 CIDR: 66.196.64.0/18 NetName: INKTOMI-BLK-3 NetHandle: NET-66-196-64-0-1 Parent: NET-66-0-0-0-0 NetType: Direct Allocation NameServer: NS1.YAHOO.COM NameServer: NS2.YAHOO.COM NameServer: NS3.YAHOO.COM NameServer: NS4.YAHOO.COM NameServer: NS5.YAHOO.COM Comment: This netblock contains Web Crawlers. Please Comment: contact *****@inktomi.com for questions or concerns. RegDate: 2001-10-30 Updated: 2003-09-26 AbuseHandle: ZI107-ARIN AbuseName: Inktomi Corporation AbusePhone: +1-650-653-2800 AbuseEmail: *****@inktomi.com TechHandle: ZI35-ARIN TechName: Inktomi Corporation TechPhone: +1-650-653-2800 TechEmail: ******@inktomi.com OrgTechHandle: ZI35-ARIN OrgTechName: Inktomi Corporation OrgTechPhone: +1-650-653-2800 OrgTechEmail: ******@inktomi.com Both of these were VERY interesting results. Anyone know what's up?
Here is some information on both: http://www.robotstxt.org/wc/active/html/xget.html http://www.robotstxt.org/wc/active/html/golem.html I don't see either of those being owned by SE's and the IP you saw may have been using a server of theirs *shrug*...
Software Platform mac Software Language HyperTalk/AppleScript/C++ Google has gone too far if they've started spidering with AppleScript.
What's going on then? Are they spoofing the IP? Contract work for the search engines? Golem's file says it does contract work. I'm confused. Googlebot hasn't touched my site in a couple days yet their cache of my site is up to date...