iamsgf
Jan 2nd 2008, 8:48 am
Hi All,
I know this is the deep dark depths of DP and few people venture to this forum, however I am hoping that somebody with knowledge will hear my plea for help and answer my call!!!!
I am having a serious issue with YAHOO making thousands of requests and causing my server to die!!
I need to set-up the robots.txt but how do i deal with dynamic URLs?
example: http://www.xxxxxxx.com/ppc.php?id=234455
I have /ppc.php in the disallow but it is still spidering all the pages. I cant use * as a wild card for anything after. What else can I do?
Obviously I dont want to block YAHOO either.
I also have a url structure of xxxxx.com/?ppc&GCIDxxxxxxxxxx that i need to block.
PLEASE HELP!
Thanks
I know this is the deep dark depths of DP and few people venture to this forum, however I am hoping that somebody with knowledge will hear my plea for help and answer my call!!!!
I am having a serious issue with YAHOO making thousands of requests and causing my server to die!!
I need to set-up the robots.txt but how do i deal with dynamic URLs?
example: http://www.xxxxxxx.com/ppc.php?id=234455
I have /ppc.php in the disallow but it is still spidering all the pages. I cant use * as a wild card for anything after. What else can I do?
Obviously I dont want to block YAHOO either.
I also have a url structure of xxxxx.com/?ppc&GCIDxxxxxxxxxx that i need to block.
PLEASE HELP!
Thanks