Why is Yahoo crawling my site to death?

Very helpful. Thanks

 

Upon further investigation.
It seems that each of those Yahoo addresses, is the main address for sub-addresses of Yahoo ISP's all around the world.

So each of them is literally 100's of ISP IP's like SBC, Net Vision, Golden Lines, Inter Business/Telcom, AOL U.K., BT Yahoo, Clarenet, and so on.
So I am no longer freakin out.
Thanks for the help.

 

That's the stats from one of my websites:



Note that the website has aroud 300 pages and MSN/YAHOO have aroud 4000 hits ... Also 90% of my traffic comes from GOOGLE and from them I got only 2000 hits ... funny ..

Cheers,

Venetsian

 

I don't have as many unknown bots, I think I have 2, but 21 total. But I see where you are coming from. My Google hits are aroud40,000!! It does look familiar, thanks

 

hahaha nice reply, yeah a worm if you want it to die use pesticides hahaha

 

They could be checking for any changes that you make on the site. hmm

 

Very nice site. Yes, it's a good thing there visiting your site just enjoy it. It's part of being successful.

 

Thanks.
Yes I am starting to understand that the particular IP's in question are in fact, friendly.

 

Understanding Search Engine Crawlers101

Most search engines use two bots to do work.

All search spiders crawl the actual server.

They do not need links to find your website.

One bot will crawl the server, pick up the pages it finds, and proceeds to send the page to it's index. (indexing bot)

Another bot will crawl all the links from the pages, to other internal pages, and external links to other websites. (crawl bot)

Bots will pick up just a few pages of your website at each visit.

The more pages you have, the longer it takes to get all pages indexed.

1,000s of pages could take a year to see them all indexed.

A hit is a section of a page

Some pages can have 10 to 50 hits, each time a spider or human visits

hits = shits

Hope this helps

 

Very nicely said Sem-Advance.

 

With much complaining about yahoo and msn bots being too active on some forums, have to ask how to bring them to mine? I.e. link in my signature here at Digital Point Forums was not enough.

I have google bot regularly wondering around but msn and yahoo bots are nowhere to be seen.

Can I do anything more with msn than to add my site at live.com? As I read, they are working on msn add sitemap service, can I add websites' sitemaps to yahoo somewhere?

 

maybe you have the meta set to scan everyday? What ever it is pls update as I would like to know