Urgent Help Please - Yahoo Keeps Killing My Site!

Discussion in 'robots.txt' started by iamsgf, Jan 2, 2008.

0
  1. #1
    Hi All,

    I know this is the deep dark depths of DP and few people venture to this forum, however I am hoping that somebody with knowledge will hear my plea for help and answer my call!!!!

    I am having a serious issue with YAHOO making thousands of requests and causing my server to die!!

    I need to set-up the robots.txt but how do i deal with dynamic URLs?

    example: http://www.xxxxxxx.com/ppc.php?id=234455

    I have /ppc.php in the disallow but it is still spidering all the pages. I cant use * as a wild card for anything after. What else can I do?

    Obviously I dont want to block YAHOO either.

    I also have a url structure of xxxxx.com/?ppc&GCIDxxxxxxxxxx that i need to block.

    PLEASE HELP!

    Thanks
     
    iamsgf, Jan 2, 2008 IP
  2. boltok

    boltok Active Member

    Messages:
    257
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    78
    #2
    This should block the dynamic URLs you want the spiders to ignore while indexing your main site:

     User-agent: *
Disallow: /ppc.php?
Disallow: /?
    Code (markup):
     
    boltok, Jan 5, 2008 IP
  3. SwapsRulez

    SwapsRulez Peon

    Messages:
    32
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Just add the below code in your robots.txt

    Cheers mate... Peace!
     
    SwapsRulez, Jan 12, 2008 IP
  4. tdave

    tdave Peon

    Messages:
    33
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    iamsgf,How your site now??
     
    tdave, Mar 11, 2009 IP