I am having trouble with Yahoo crashing my server each time it visits!! I have a few thousand keywords being used for PPC campaigns with tem, and their system keeps coming to the site to varifiy the urls for the campaigns, but the thing is it does not do it bit by bit, but it really hits it hard and crashes my site. Sometimes the server can be going up and down for 12 hours!! If I block yahoo then my PPC campaigns will be rejected. I found a couple of links about this (http://www.webmasterworld.com/forum33/2887.htm) but could not find a solution. Does anyone have any ideas? Thanks
Do you have a web hosting company? If your server falls down so many times for so long then it's not good enough...
finally I have identified the problem!!!!!! Each time a PPC campaign is activated Yahoo check all the links to confirm that they are valid (I currently have 100,000+ keyword campaigns). Yesterdaythe company in charge of the PPC reloaded all of the campaigns in one go, therefore Yahoo tried to varify everything in one go...... basically it was not far off a DOS attack on my server. Due to the stupid logic of Slurp, it keeps trying the links until it finds them, therefore it just killed my server and everytime it stood up again it kicked it straight back down trying to check links!! A Royal Pain In The Ass!!!!
You can try to limit open/active connections from one source IP. However, your landing pages may be reported as broken if you kick the Slurp It's said that some people @ Y don't have a clue how to fix such an easy problem actually..... Try also the following in robots.txt, does it make a difference? User-agent: Slurp Crawl-delay: 0.25