What's the highest server load you've ever seen?

Discussion in 'Site & Server Administration' started by gemini181, Jul 17, 2007.

  1. #1
    • Server load depends on many factors
    • A 'high number' is not always bad

    At some point the load can really slow down a web server.
    What's the highest server load you've ever seen?
    How many CPU's?

    (((Note: This is not a major host; they are usually way under 3.5 (4 CPU), I hope it's just a 'rare spike' :eek:)))

    [​IMG]

    Edit: The server totally crashed and came back up pretty fast.
    Server Load 1.48 (4 cpus)
    Still not a great day for a formerly above average host.
     
    gemini181, Jul 17, 2007 IP
  2. cormac

    cormac Peon

    Messages:
    3,662
    Likes Received:
    222
    Best Answers:
    0
    Trophy Points:
    0
    #2
    All hosts will experience something like this and it can for many different reasons. We get a lot of it due to some high end customers that we host.

    Its unfortunate when the reason is due to a site on a shared server that ends up running some crazy script, that a more common cause.
     
    cormac, Jul 17, 2007 IP
  3. login

    login Notable Member

    Messages:
    8,849
    Likes Received:
    349
    Best Answers:
    0
    Trophy Points:
    280
    #3
    My host had about 20 in server load on a 4 cpu server because of me putting proxies on it. The server was still running smooth.
     
    login, Jul 17, 2007 IP
  4. gemini181

    gemini181 Well-Known Member

    Messages:
    2,883
    Likes Received:
    134
    Best Answers:
    0
    Trophy Points:
    155
    #4
    117 is awful high.
    The server did crash, or at least it was 100% unaccessible for a few minutes.
    ~~~~~~~
    Q) Can a modern Web server crash, be re-booted, and all back up (FTP, CPanel, DB, www, etc) in under 3 minutes?
    ~~~~~~~
     
    gemini181, Jul 17, 2007 IP
  5. cormac

    cormac Peon

    Messages:
    3,662
    Likes Received:
    222
    Best Answers:
    0
    Trophy Points:
    0
    #5
    3 minutes is almighty fast and it really depends on the manufacture & model of the server. For example a HP DL140 can take up to 4 minutes just to boot as it loads RAID controllers etc.

    Although its not impossible...
     
    cormac, Jul 18, 2007 IP
  6. ilxeon

    ilxeon Peon

    Messages:
    336
    Likes Received:
    48
    Best Answers:
    0
    Trophy Points:
    0
    #6
    even server load of 3.5 (4) is not considered good as it degrades Mysql performance.
    Infact that load meter is not CPU load but system load and it does not determine CPU usage. Load could peak because of you drive not able to server data at the same pace and not because CPU is underpowered. But still high system load is indicative of something not right.
     
    ilxeon, Jul 18, 2007 IP
  7. InFloW

    InFloW Peon

    Messages:
    1,488
    Likes Received:
    39
    Best Answers:
    0
    Trophy Points:
    0
    #7
    The server load # you need to consider the # of cpu's on the system and various other pieces of hardware.

    Most people figure 2*number of cpu's is an acceptable load average before you see a noticable slowdown. The reason for this is because the load average is actually the number of processes waiting to run. So my load average of 4 on my quad cpu system each process is handled as they come in.

    So using that theory I can run a bunch of backup processes as a nice value on a system that is high load already. The nice processes will use CPU remaining to attempt to finish their task. Now what happens is the load average will start going up because those processes are waiting and the load average is measured by # of process in queue to run.


    Now the highest load average I've been able to recover was around 700. The machine ran out of CPU on another process causing the possible 500 httpd processes to start sitting waiting producing a load of 500. Then various other services kicked in trying to grab CPU as well causing it to continue to go up and processes were added to the queue. Managed to get into the box and run a few commands and knock the load average back down. The highest I've seen on a power overall that I had to reboot was 1500 and it was pretty dead.


    Now on the subjects of reboot times it really vary's depending on your server setup. I've had servers reboot in just a few mins while others with IPMI cards take up to 10 minutes to be booted up that you can get into SSH.
     
    InFloW, Jul 18, 2007 IP
  8. agnivo007

    agnivo007 Peon

    Messages:
    4,290
    Likes Received:
    289
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Well, my AMD XP 2000+ box reboots in under 2 mins (with plesk) ;)
     
    agnivo007, Jul 18, 2007 IP
  9. mixtmobile

    mixtmobile Peon

    Messages:
    159
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    My host had about 20 in server load on a 4 cpu server because of me putting proxies on it. The server was still running smooth.
     
    mixtmobile, Jul 18, 2007 IP
  10. nddb

    nddb Peon

    Messages:
    803
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #10
    That's pretty high, but for a single CPU machine, that's only 25 load per CPU (obviously VERY bad), but I've seen mine go to 500 for a SINGLE CPU and lock the box due to a bad mod_rewrite rule that created an infinte loop. (never use the flag they tell you not to use, even if you are sure you know what you're doing! LOL)
     
    nddb, Jul 20, 2007 IP
  11. nddb

    nddb Peon

    Messages:
    803
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #11
    I don't think load considers drive speed. Load is how much CPU utilization. For instance, if it's 1, then the CPU is at full capacity. At a lot of 2 then 50% of processes are waiting for the CPU to finish with the first set. (iirc)

    It's just a measure of what processes are waiting for the CPU to be free. So, basically... yea... it does measure CPU usage.

    Load is a timed sample of CPU usage, basically. I don't think it considers memory usage, or drive i/o or anything else, although they probably correlate, depending on what you are doing.

    But anyone who has used a fork bomb knows it's not necessarily how much memory you use, just how busy you keep the CPU, if it can't keep up, load goes over 1.0 (for one CPU) or 2.0 (for 2 CPUs) etc...
     
    nddb, Jul 20, 2007 IP
  12. InFloW

    InFloW Peon

    Messages:
    1,488
    Likes Received:
    39
    Best Answers:
    0
    Trophy Points:
    0
    #12

    Load does factor in other things and this is how:

    The load average is the number of processes currently in queue for CPU time. Now there is more than one way to cause CPU usage issues.

    Cpu0 : 0% us, 0% sy, 0.0% ni, 99.0% id, 1.0% wa, 0.0% hi, 0.0% si

    us is actual user cpu, sy is cpu taken by the system, ni is taken by nice processes, id is idle time and wa is wait, hi and si I cannot remember.

    Now why is this important? wa is i/o wait so the CPU waiting for hard drive time and thus it is causing it to be less than efficient. So slow hard drives cause load to go up because the CPU is being used less. Same respect lack of ram will cause swapping thus making the cpu inefficient.

    So load average does factor in the other aspects due to the fact the CPU relies on them.
     
    InFloW, Jul 20, 2007 IP
    gemini181 likes this.
  13. nddb

    nddb Peon

    Messages:
    803
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #13
    It all correlates, but the end result is the CPU has to wait, regardless of the factors, it's a measure of CPU has to wait. Usage over a specific period of time. I was simply saying, it is definitely is a measure of CPU usage.

    I mean, the factors correlate, but swapping is an OS problem, not a CPU problem. The CPU doesn't write/read data to an HD. A program utilizing the CPU may have to wait for the hard drive, causing it to run longer and drive load up, but that load is CPU load. Because the program makes the CPU wait.

    So, yes, there are factors that contribute, but load is only a measure of CPU wait time. The rest is OS, or apps, or whatever is running on the CPU.

    So load shows you the symptoms, but doesn't tell you the exact effect. Like I said, it is a measure of CPU wait time. Drive speed, userland apps, the OS may affect that wait time, but that is not what load measures. Drive speed and memory usage are not _directly_ considered when calculating load. Only CPU wait time.

    --

    So yea, I think we are basically agreeing. Drive delays and other things can cause applications or the OS to run longer and drive up load. But I'm just differentiating that load is not a DIRECT measure of those things, possibly indirect, but you can't determine the cause of high load simply by seeing high load. All you know is the CPU is very busy.
     
    nddb, Jul 20, 2007 IP
  14. gemini181

    gemini181 Well-Known Member

    Messages:
    2,883
    Likes Received:
    134
    Best Answers:
    0
    Trophy Points:
    155
    #14
    There is a lot of good info here
    (which I didn't see until today)
    ~~
    Now I know much more about server load :)
     
    gemini181, Aug 7, 2007 IP