Daily, time to time cpu load problem

Discussion in 'Site & Server Administration' started by qweqwe, Aug 9, 2007.

  1. #1
    I have a dedicated server specs: AMD 3500+ 64 Bit CPU, 1 GB Ram, 160 GB Sata Drive. For 1 month, CPU load average reaches 40-50 value. This happens about 5-6 times in a day. When I stop httpd service for 30 seconds everything goes normal. I think this is not a DoS attack because it comes systematic, I dont believe no one makes this regularly except bots.
    Maybe its a system service or a cronjob but it stops when I turn off httpd service?
    How can I be sure about what's making this regularly load?

    I also did set up a script which mail me when load average of system goes crazy and restart httpd service. But instant restart is not working to stop load increase. Any idea to stoping and delaying 30 seconds before restart a service.

    I need to figure out this problem please help me to find out.
     
    qweqwe, Aug 9, 2007 IP
  2. cyanide

    cyanide Peon

    Messages:
    483
    Likes Received:
    26
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Did you run TOP to find out what's causing it?

    You'll need to have a command prompt (shell) open all day with top running to catch what is causing the load.

    What kind of sites are you running?
    Has your server gone through a security audit and lock-down?
     
    cyanide, Aug 9, 2007 IP
  3. agnivo007

    agnivo007 Peon

    Messages:
    4,290
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Maybe it's your stats updating or some cron jobs running?
     
    agnivo007, Aug 9, 2007 IP
  4. qweqwe

    qweqwe Peon

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    cyanide: Yes I always check TOP, actually SSH window always open to see server status. TOP screen full of httpd threads like these:
    img65.imageshack.us/img65/838/topscreenqu1.jpg

    3 high traffic sites in this server, one of theme is an image hosting site. Hotlink prevention enabled. Whole server getting 20-25.000 unique visitors. Yes I know it's high but server can handle 99% of a day except these high load times.

    What do you mean that "security audit"? Sorry Im not a pro about this server and linux subject.

    agnivo007: If its stats it stops when I stop httpd service? And how can I be sure and what am I gonna check?
     
    qweqwe, Aug 10, 2007 IP
  5. cyanide

    cyanide Peon

    Messages:
    483
    Likes Received:
    26
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Hi,

    When I say security audit... anytime a new server goes online, it needs to be hardened and locked down with security measures....
    If this has not been done, who knows what's going on.... maybe you've got some hacks running...

    When was the last time your server was re-booted?
    Did you examine the apache and mysql logs?
    Also, how much ram and swap is being used at these times?
     
    cyanide, Aug 10, 2007 IP
  6. qweqwe

    qweqwe Peon

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    It has been 13 days since last reboot.

    Here is the TOP screen when load starts:
    img172.imageshack.us/img172/5236/top1yy6.jpg

    A few minutes lates server load average goes 10:
    img167.imageshack.us/img167/114/top2tz6.jpg

    I didnt see anything weird in log files but actually I dont know what to investigate in it.
     
    qweqwe, Aug 10, 2007 IP
  7. bouncer69

    bouncer69 Peon

    Messages:
    978
    Likes Received:
    43
    Best Answers:
    0
    Trophy Points:
    0
    #7
    I had similar problem with one of my box in fasthost.co.uk. I didn't get any solution also I had contacted with support team no solution. Later I moved my website to another new server. Now no more problem.
     
    bouncer69, Aug 10, 2007 IP
  8. qweqwe

    qweqwe Peon

    Messages:
    5
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    It is last thing to do. Moving whole server is a torture.
     
    qweqwe, Aug 10, 2007 IP
  9. agnivo007

    agnivo007 Peon

    Messages:
    4,290
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Well, I've seen load issues with particular kernel versions. Sometimes memory usage and cpu uage is reported wrongly as the kernel release might be buggy.
     
    agnivo007, Aug 11, 2007 IP