yahoo slurp invasion

[mpea]

I’m getting crawled too much by yahoo slurp:
lj511137.crawl.yahoo.net

it seems to be downloading 1/2 a gig a day, more than 700 visits per day... its killing me

I get about 20 user 10 visits form google and 700-1000 visits from yahoo, what the hell is going on!!!!!

I have tried restricting it with a robots text yesterday, but hasn’t had much of an affect today,

User-agent: Slurp
Crawl-delay: 2

How long does it take to kick in? I don’t want to ban it completely, but if its going to have a negative affect, I’m might have to ban yahoo omg :(

[sezerb]

That does seem a little excessive. Maybe wait a day or 2 for the robots.txt to work.

[Dr. XYZ]

Anyway if you don't want to limit Yahoo Slurp there is help available on-site because many people complain of this robot.

Pay a visit to Yahoo to find out more.

·

[reapr]

Yeah just block it totally! just kidding ... you can limit it's crawl rate in your robot.txt file plenty of tutorials around on it. Good luck.

[mpea]

yup, thats what i did with this:
User-agent: Slurp
Crawl-delay: 2
Just wasn’t sure how long it would take until yahoo stopped misbehaving

[dairyman]

Just wait and watch! It will take some more time for the crawling frequency to be restricted.

[enous]

HOHO..maybe it crazy....LOL