Mobile Phones - Boxing Class - Adverse Credit Remortgage - Loans - Mortgages

PDA

View Full Version : Urgent Help Please - Yahoo Keeps Killing My Site!


iamsgf
Jan 2nd 2008, 8:48 am
Hi All,

I know this is the deep dark depths of DP and few people venture to this forum, however I am hoping that somebody with knowledge will hear my plea for help and answer my call!!!!

I am having a serious issue with YAHOO making thousands of requests and causing my server to die!!

I need to set-up the robots.txt but how do i deal with dynamic URLs?

example: http://www.xxxxxxx.com/ppc.php?id=234455

I have /ppc.php in the disallow but it is still spidering all the pages. I cant use * as a wild card for anything after. What else can I do?

Obviously I dont want to block YAHOO either.

I also have a url structure of xxxxx.com/?ppc&GCIDxxxxxxxxxx that i need to block.

PLEASE HELP!

Thanks

boltok
Jan 5th 2008, 1:39 am
This should block the dynamic URLs you want the spiders to ignore while indexing your main site:

User-agent: *
Disallow: /ppc.php?
Disallow: /?

SwapsRulez
Jan 13th 2008, 12:19 am
Just add the below code in your robots.txt

User-agent: yahoo
Disallow: /ppc.php?

Cheers mate... Peace!