I'm getting a little fed up with Yahoo. They're spidering my sites like crazy, eating up my bandwidth but I don't get a single hit from their search??? I'm running Google Analytics and two stats packages on my server to analyze where my traffic is coming from but there's nothing from Yahoo. I have numerous first page keywords in Google and I get a ton of traffic from both Google and MSN/Live, but absolutely nothing from Yahoo. Besides those two search engines, I get the most traffic from DigitalPoint (thanks). Has anybody had this problem with Yahoo? Excessive constant slurping of your sites but no traffic? I'm about ready to block Yahoo in my robots.txt, what's the point of letting them spider my sites if they don't turn any traffic or index me. As far as the Yahoo Directory, don't *ever* waste your money on it. I wasted $299 two years ago and my site still doesn't even turn up for my keywords (but sites that are 404 or no longer working turn up in the top listings).
I have the same issue, however bandwidth and server loads are not an issue for me. I do receive some visitors from Yahoo across my sites, but it is minimal compared to Google. Regarding the Yahoo Directory.... The Yahoo Directory is a trusted directory, and its been around for a while. Some will argue (including myself) that being listed in a trusted directory helps your serps in other search engines. I have sites listed in Yahoo's Directory, and I receive no traffic from those listings. But one thing I have noticed across multiple sites is that my sites climb in the serps (especially Google) after getting listed in the Yahoo Directory. Yahoo is number 2 in the search engine market for a reason. They have some work to do if they want to catch Google. But don't count them out yet... MSN/Live just had an algo update that made some happy and others sad. Yahoo is bound to do the same, and you don't want to block their bot when it happens. If bandwidth and/or server loads are an issue, shop for a better host.
How noticed many webmaster has the same thing. http://forums.digitalpoint.com/showthread.php?t=507758 Thanks for your experience. we are no necessry cost $300 for submission to directory.
I have two very powerful dedicated servers with PEER1, bandwidth is definitely not an issue. It's a question of resource economics. I'm an "old school" Unix C/C++ programmer who believes in "not wasting a byte" if you have to. That includes my bandwidth, Apache processes, database connections, etc. I primarily run data driven sites, so a wayward spider is not only using my bandwidth, it's using server process and databse resources. If there's nothing to gain from it, then it's a waste. Waste not, want not. You may be right, I've recently seen a surge in traffic from MSN/Live, so maybe Yahoo is spidering like crazy for a new update. I guess I'll just wait and see what happens. thanks for the input snowbird.
You can add a crawl delay to your robots.txt to tell Slurp to slow down (but not stop completely). http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-03.html
I'm not so worried about the Yahoo spiders now because I'm seeing a ton of my pages indexed in Yahoo and finally starting to see some traffic from them. It looks like Yahoo has been doing some serious indexing.
Yahoo uses different IP's for different pages so you see a ton of them. Google uses the same IP generally so you only see the one crawling page after page. Same result different method. Though Yahoo seems to be broken today.
Yes, they have stopped spidering so intensely. It was only happening for a few weeks but I'm actually geting hits from Yahoo now and the sites that were being spidered are showing up under some keywords in Yahoo now.
Yahoo uses about 15x as much bandwidth as the next heaviest hitter, MSN. They send a bit of traffic, about 2x or 3x MSN, but a fraction (that I haven't calculated) of Google. I understand your resource economics dilemma, and I feel the same way myself. If bandwidth hadn't gotten so cheap, I would probably ban Yahoo in my robots.txt, but as they do send me a bit of traffic, and the cost is close enough to nil, I just ignore it. Definite agreement on their directory, by the way. People would get more mileage out of taking their wife to a fancy dinner than paying for a slot in the Yahoo Dir.