minstrel
Apr 1st 2007, 5:04 pm
Yahoo! Search Crawler, Slurp, is moving (http://p7.hostingprod.com/@www.ysearchblog.com/archives/000426.html)
Anyone who has seen their web server logs, has seen the Yahoo! crawler come by and leave its mark. It identifies itself as 'Yahoo! Slurp' and its domain is inktomisearch.com. For those of you who follow the search space, you can easily guess that this is one of the last historic remnants from our acquisition of Inktomi a few years ago. Well, the crawler finally decided to move and find a new home. We are moving our crawler from inktomisearch.com to crawl.yahoo.net.
The user-agent will continue to be Yahoo! Slurp, hence you do not need to make any changes to your robots.txt file. Nor are we changing the actual IP addresses of the crawler infrastructure during this transition. However, if you do any reverse DNS checks for the crawler identity or have any network access rules to allow inktomisearch.com, please also update them to allow for crawl.yahoo.net.
The crawlers will be switched over in phases starting immediately. You'll see an increasing number of Yahoo! Slurp visits from crawl.yahoo.net and fewer and fewer from inktomisearch.com during the transition, which will take a few weeks.
Keep an eye on this space and the Yahoo! Search Blog for more details on when the migration is complete and details on bot verification with the new domain name.
Priyank Garg
Product Manager, Yahoo! Search
Anyone who has seen their web server logs, has seen the Yahoo! crawler come by and leave its mark. It identifies itself as 'Yahoo! Slurp' and its domain is inktomisearch.com. For those of you who follow the search space, you can easily guess that this is one of the last historic remnants from our acquisition of Inktomi a few years ago. Well, the crawler finally decided to move and find a new home. We are moving our crawler from inktomisearch.com to crawl.yahoo.net.
The user-agent will continue to be Yahoo! Slurp, hence you do not need to make any changes to your robots.txt file. Nor are we changing the actual IP addresses of the crawler infrastructure during this transition. However, if you do any reverse DNS checks for the crawler identity or have any network access rules to allow inktomisearch.com, please also update them to allow for crawl.yahoo.net.
The crawlers will be switched over in phases starting immediately. You'll see an increasing number of Yahoo! Slurp visits from crawl.yahoo.net and fewer and fewer from inktomisearch.com during the transition, which will take a few weeks.
Keep an eye on this space and the Yahoo! Search Blog for more details on when the migration is complete and details on bot verification with the new domain name.
Priyank Garg
Product Manager, Yahoo! Search