Inktomi Slurp

Ok I know this is a way big duh moment. But after reading my awstats on a site that I have. I wanted to know what one of the robot/spiders were. It lists Inktomi Slurp and of course Googlebot and MSNBot. But didnt realize that Inktomi Slurp was the Yahoo's version of bots/spiders. I have so many more crawls via Inktomi Slurp than Google and MSN combined.

Does anyone know why Inktomi Slurp crawls a site often and so much? And why so many backlinks etc?

Brief stat info for the site.
Inktomi Slurp: 998
GoogleBot: 183
MSNBot: 146

Did a site:www.mydomain.com in Yahoo and it pulled up 706 pages. And at Google: site:www.mydomain.com and got 22. MSN was (lol) 17.

The site is Google PR4. And gets roughly 100+ uniques a day. I dont think it matters about PR and or uniques. But thought to cover that basis just in case.

 

possibly just yahoo trying to catch up to google anyway they can...and figuring getting more pages indexed might make people like them more...

(just a possible idea...lol)

 

It's not consistent.

I have Google pounding my web sites daily, and Yahoo slowly meanders them.

No clue why.

 

for me yahoo rarely visits my sites

 

yahoo slurp is lazy and msn is super lazy ...

 

eh? How can they be lazy did you look at the stats for my site?

I would say they may be lazy for you. But not on my website compared to Google and MSNbots.

Just wondering if anyone could answer: Does anyone know why Inktomi Slurp crawls a site often and so much? And why not so active in others, like others stated in this thread, referring to it being lazy.

 

HOw do you know that? This is the first time I hear about Inktomi Slurp.

 

Same cases for me, Inktomi Slurp will crawl my website much more than Google and MSN.

 

Is it normal to have over 100 slurp bots on your site perday? it seems most of my visits are from yahoo.

 

I get an average of 300/400 hits a day from Slurp on a site with 5000 pages
100 bots at the same time are a bit too much, but it happened on my site as well.
I get also about 100 from G
The lazy one is MS bot: today only 6 pages crawled.

 

Ahhh nothing like a little search in the Google toolbar for yahoo slurp = inktomi slurp
http://www.webmasterworld.com/forum35/3620.htm
http://www.internetadsales.com/modules/wfsection/article.php?articleid=2200
http://forums.digitalpoint.com/showthread.php?t=25764 - dated: Aug 23rd 2005
http://forums.searchenginewatch.com/showthread.php?threadid=6496

http://www.web-page-optimization.com/spiders.html - more detailed info

 

I get more hits from google then from yahoo. Not by much thou a couple of hundred hits. Live is catching up with them thou.

 

For my sites it seems that Yahoo is much slower than MSN.

 

You must rank very high at msn since msn seems not to be crawling any other site but yours.

 

If you were referring to me. Nope I rank high at Yahoo, than Google, than MSN

 

Visit www.inktomi.com and I think its pretty obvious that it's owned by Yahoo.

 

I've seen inktomi and yahoo slurp but not a inktomi slurp.
My site seems to get more Yahoo and inktomi spiders than they do from google.

 

Yahoo Slurp and MSNbot are much more aggressive in their spidering rates than Googlbot.

You can slow them down (at least theoretically - I'm not convinced Yahoo actually respects this) by ading the following to the end of your robots.txt file, placed in the root folder of your website:

User-Agent: msnbot 
Crawl-Delay: 10 

User-Agent: Slurp 
Crawl-Delay: 10 

Code (markup):

The above code is specific to MSN's spider, "MSNBot", and Yahoo's spider, "Slurp", and instructs the spiders to wait the specified amount of time, in seconds (10 seconds above, default is 1 second if not specified) before requesting another page from your site. MSNBot and Slurp have been known to index some sites very heavily, and this allows webmasters to slow down their indexing speed.

Googlebot is normally better behaved by default and so doesn't require this.

 

Okay, here is the question: Why would you want to stop the search engines from indexing your site? It's good that they index your site? Right.

 

We're not talking about STOPPING them... just slowing them down when they get bandwidth-greedy.