We have been reading the threads here and a few other users have the same issue,seems that its hard to get yahoo to crawl but its harder to control it once there. Checking our Swapshop web stats we see that yahoo stays on our site 24hr a day???? Robots.txt tells Slurp to slow down but really want to know why its hammering us and chewing the b/w? lj601545.inktomisearch.com lj602099.inktomisearch.com lj601442.inktomisearch.com UNKNOWN-72-30-61-177.yahoo.com lj601443.inktomisearch.com lj2365.inktomisearch.com The list goes on each one is 1 hit 45 secs apart The UNKNOWN-72-30-61-175.yahoo.com 254 hits lt100005.inktomisearch.com 113 We have around 2200 pages and this grows every day being a classifieds site. Is this normal? Yahoo Yahoo 18101 74.25% Google Google 3380 13.86% MSN MSN 1771 7.26% Any advice would be great thanks
this is normal as i see the same thing on my sites as well as my work sites! If you don't like em them just robot them out (and robot them in a week a month)
Slurp took down one of my shared accounts earlier in the week...we had to put a delay on it just to moderate things; it was running hundreds of simultaneous queries all over the place. What a hunk of junk it has become.
Heres some stats for you Top 50 Robots Yahoo 20820 75.96% Google 3494 12.75% MSN 1892 6.90% Gigablast 629 2.29% WiseNutBot 177 0.65% Ask Jeeves 101 0.37% Robot 59 0.22% Majestic-12 43 0.16% PicSearch 26 0.09% FindLinks 22 0.08% Java 21 0.08% libWWW 17 0.06% Yahoo is is just simply out of control. Is it something we are doing or coded? Does any one else ban yahoo in robots.txt for a few days or just leave it? It starts costing us bandwidth every month and we only see 1-2 hits a day compared with google at 150+ a day. Any one have sucess in controlling yahoo? Currently we hit up to 5000 hits a day then it goes to 2000+ a day then its back again but it just gets hungier every time. Any advice would be great!
I think so too... although I do feel Slurp's overdoing it... It hits my site 4 times more than MSNbot... but I get equal hits from both these SE.
Dont get me wrong our issue is that we are on the biggest plan at 14gb /month Our site does up to 3000-5500 hits a day. Yahoo 20820 75.96% But for all this traffic we dont get more that one or two hits from yahoo. Google returns 100 fold more? I know its good to get a SE like slurp thru your site and really we have most of the top 10 SE weekly but what is the point to crawl your site and not return many hits? Keywords, description works on google and msn 80% but yahoo ???
"slurp" will gradually slow down after couple of weeks once it done with its "Yahoo Dance" and don't get excited if lots of your pages get indexed... they will drop after final re-shuffle..
Get a better server. 14gb per month ain't squat. For a few bucks per month more you can get 50gb or 75gb. Build more pages, add more content, do it as fast as possible and let yahoo slurp the hell out of it. After google screws you you'll be glad you've got thousands of pages in yahoo.
I also agree with wibr.. you'll be glad you're in yahoo while google thinks about things in good time...
Well in Kiwi land we are on one of the biggest plans. Next one up is a heck of a big step. I guess some of you are lucky to have cheap plans..... We dont wish to host external to NZ since this will slow the site done hence where our issues start. Anyway..... Our point was Yahoo is crawling 80% mor than google yet google returns 20:1 hits and yahoo doesnt. Whats the point of yahoo if it doesnt return hits. Yahoo is on our site 24/7 we have slowed it down to allow it to crawl. Its crawling foot pint is strange. A hit every minute, and then after a week or so a big crawl from all the different Yahoo crawlers. To block Yahoo is mad the 2nd biggest or maybe soon the biggest SE. For a while we though we had a issue with trapping Slurp but others report the same issue.
Yahoo! Slurps alot on my site as well. Right now, I have 21 bots from them on my site. 21 from Yahoo! alone.