Mediapartners-google bot and other bots causing 500 errors

Discussion in 'Site & Server Administration' started by channel5, Jul 3, 2008.

0
  1. #1
    Hi

    I have a site written in JSP, and for some reason I am finding that Mediapartners-google bit is getting an HTTP 500 Error 50% of the time.. other bots such as Baidu and some more malicious ones are also getting 500 Errors.

    Has anyone else seen such an effect with bot's causing crashes, but not normal traffic?

    Any help or comments appreciated.
     
    channel5, Jul 3, 2008 IP
    calum likes this.
  2. calum

    calum Peon

    Messages:
    2,821
    Likes Received:
    141
    Best Answers:
    0
    Trophy Points:
    0
    #2
    If you check the urls the bots get the internal server errors are, do they also give you errors when you visit them in a browser?
     
    calum, Jul 3, 2008 IP
  3. channel5

    channel5 Peon

    Messages:
    2
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Hi Callum :)

    If I go to the URL's that the Mediapartners-google bot gets a 500 error on i see the pages fine.

    I also find in the logs that after a 500 error the bot tries the URL again and gets a successful 200 result,,

    It's really strange
     
    channel5, Jul 5, 2008 IP
  4. calum

    calum Peon

    Messages:
    2,821
    Likes Received:
    141
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Sounds like a permission or resource problem.

    Do you know if you are on a phpsuexec server? To find out get a phpinfo() file and find where it says "Server API". If it says CGI you are on a phpsuexec server. If you are, try lowering the permissions of the files that are giving errors (eg. so if they are 0777 change them to 0644).

    Also, do you use shared or vps/dedicated hosting? It may be that the server is out of memory or hitting a cpu limit.
     
    calum, Jul 5, 2008 IP