Help! Server load increasing from ~5 to ~200% (4 cpus)

Hello,

I'am using Wordpress 2.9. Webserver 8GB from Hostgator.

I'am fighting with this problem for long time but still can not find the solution. Php switched to run as an apache module, Php 5 in DSO, Apache suEXEC, eaccelerator disabled, but this configuration started making huge server load on server.

Server load increasing from 1 to 250 (4 cpus) and server stops, after period of time its back again and stops in about 10 minutes. What can make this problem and how can I fix it?

Thank you guys.
Any ideas appreciated!!!

 

What does your output of 'top' look like? Have you done any MySQL or apache optimization? Perhaps you are under a dos attack.....

 

When its working (slowly) server load is being about 30% (4 cpus) my top looks like this:

top - 12:51:13 up 11:05, 1 user, load average: 30.74, 30.84, 24.33
Tasks: 517 total, 37 running, 479 sleeping, 0 stopped, 1 zombie
Cpu(s): 93.3%us, 4.9%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 1.8%si, 0.0%st
Mem: 8310280k total, 6987016k used, 1323264k free, 103332k buffers
Swap: 5421896k total, 0k used, 5421896k free, 885004k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26351 nobody 15 0 47248 29m 4312 R 43.9 0.4 0:03.33 httpd
26297 nobody 16 0 48536 30m 4312 R 41.7 0.4 0:03.21 httpd
26018 nobody 15 0 48544 30m 4312 S 30.7 0.4 0:03.39 httpd
25597 nobody 16 0 45972 28m 4320 R 26.3 0.4 0:04.81 httpd
25604 nobody 16 0 49816 32m 4316 R 26.3 0.4 0:08.23 httpd
25946 nobody 16 0 43672 26m 4316 R 24.1 0.3 0:05.28 httpd
26328 nobody 16 0 45728 28m 4320 R 24.1 0.4 0:03.22 httpd
26642 nobody 16 0 43936 26m 4316 R 17.6 0.3 0:01.10 httpd
26422 nobody 16 0 48544 31m 4320 R 13.2 0.4 0:01.80 httpd
24606 nobody 16 0 47864 30m 4908 R 11.0 0.4 0:12.15 httpd
24796 nobody 16 0 48544 30m 4324 R 11.0 0.4 0:13.85 httpd
25535 nobody 15 0 48536 30m 4320 S 8.8 0.4 0:08.71 httpd
26412 nobody 16 0 43924 26m 4304 R 8.8 0.3 0:00.88 httpd
26416 nobody 16 0 47804 30m 4308 R 8.8 0.4 0:01.45 httpd
26590 nobody 16 0 48012 30m 4320 R 8.8 0.4 0:01.39 httpd
24627 nobody 16 0 47016 30m 4936 R 6.6 0.4 0:19.66 httpd
25471 nobody 15 0 48020 30m 4324 R 6.6 0.4 0:09.13 httpd
26239 nobody 15 0 48020 30m 4308 R 6.6 0.4 0:03.23 httpd
26586 nobody 16 0 39292 22m 4296 R 6.6 0.3 0:01.02 httpd
22309 root 15 0 2564 1352 796 R 4.4 0.0 0:17.74 top
23398 nobody 16 0 43692 27m 4940 R 4.4 0.3 0:21.17 httpd
25657 nobody 15 0 48024 30m 4320 S 4.4 0.4 0:04.16 httpd
25919 nobody 15 0 38996 22m 4328 S 4.4 0.3 0:08.86 httpd
25984 nobody 16 0 43680 26m 4316 R 4.4 0.3 0:05.89 httpd
25997 nobody 15 0 43928 26m 4316 S 4.4 0.3 0:05.15 httpd
26371 nobody 16 0 43932 26m 4300 R 4.4 0.3 0:00.91 httpd
26631 nobody 15 0 39252 22m 4248 S 4.4 0.3 0:00.34 httpd
25071 nobody 15 0 39032 22m 4608 S 2.2 0.3 0:14.51 httpd
25098 nobody 15 0 43692 27m 4936 S 2.2 0.3 0:12.68 httpd
25634 nobody 15 0 45976 28m 4320 S 2.2 0.4 0:07.19 httpd
25962 nobody 15 0 45992 28m 4312 R 2.2 0.4 0:04.50 httpd
26302 nobody 16 0 48824 31m 4320 R 2.2 0.4 0:03.02 httpd
26305 nobody 15 0 45972 28m 4320 S 2.2 0.4 0:02.32 httpd
26369 nobody 15 0 46020 28m 4348 S 2.2 0.4 0:02.95 httpd
26394 nobody 15 0 39540 22m 4268 S 2.2 0.3 0:00.68 httpd
26398 nobody 15 0 48060 30m 4312 R 2.2 0.4 0:02.52 httpd
1 root 15 0 2072 640 540 S 0.0 0.0 0:00.99 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.01 migration/0
3 root 34 19 0 0 0 R 0.0 0.0 0:00.03 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/1


When server stops it looks like this:

top - 18:57:01 up 2 days, 17:51, 4 users, load average: 205.34, 112.81, 59.86
Tasks: 408 total, 259 running, 149 sleeping, 0 stopped, 0 zombie
Cpu(s): 99.9%us, 0.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8310280k total, 3000696k used, 5309584k free, 281032k buffers
Swap: 5421896k total, 50136k used, 5371760k free, 1809580k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24604 nobody 23 0 47496 13m 4136 R 3.3 0.2 0:04.49 httpd
24605 nobody 23 0 52852 25m 10m R 3.3 0.3 0:04.54 httpd
24606 nobody 25 0 52852 25m 10m R 3.3 0.3 0:04.81 httpd
24607 nobody 24 0 47496 13m 3868 R 3.3 0.2 0:04.60 httpd
24609 nobody 25 0 47496 13m 3868 R 3.3 0.2 0:05.60 httpd
24614 nobody 25 0 53172 27m 11m R 3.3 0.3 0:06.00 httpd
24616 nobody 23 0 52852 25m 10m R 3.3 0.3 0:04.59 httpd
24618 nobody 25 0 52864 26m 11m R 3.3 0.3 0:05.69 httpd
24621 nobody 25 0 52852 26m 11m R 3.3 0.3 0:04.80 httpd
24622 nobody 20 0 52936 26m 11m R 3.3 0.3 0:01.89 httpd
24623 nobody 25 0 52852 25m 10m R 3.3 0.3 0:05.69 httpd
24624 nobody 25 0 52860 25m 10m R 3.3 0.3 0:04.79 httpd
24625 nobody 25 0 52864 26m 11m R 3.3 0.3 0:04.79 httpd
24627 nobody 25 0 52840 25m 10m R 3.3 0.3 0:04.81 httpd
24639 nobody 23 0 47496 13m 3880 R 3.3 0.2 0:04.49 httpd
24640 nobody 25 0 53676 27m 11m R 3.3 0.3 0:04.99 httpd
24641 nobody 25 0 52852 26m 11m R 3.3 0.3 0:04.83 httpd
24642 nobody 22 0 52856 26m 11m R 3.3 0.3 0:02.51 httpd
24643 nobody 25 0 52852 26m 11m R 3.3 0.3 0:04.80 httpd
24644 nobody 25 0 52864 26m 11m R 3.3 0.3 0:03.00 httpd
24646 nobody 25 0 53656 26m 10m R 3.3 0.3 0:04.09 httpd
24647 nobody 22 0 52852 26m 10m R 3.3 0.3 0:02.99 httpd
24648 nobody 25 0 52864 26m 11m R 3.3 0.3 0:03.13 httpd
24650 nobody 20 0 52852 25m 10m R 3.3 0.3 0:01.80 httpd
24656 nobody 21 0 47496 13m 3872 R 3.3 0.2 0:03.71 httpd
24658 nobody 21 0 52856 26m 11m R 3.3 0.3 0:04.19 httpd
24662 nobody 25 0 52852 25m 10m R 3.3 0.3 0:04.74 httpd
24665 nobody 22 0 52840 25m 10m R 3.3 0.3 0:04.20 httpd
24667 nobody 25 0 52864 26m 11m R 3.3 0.3 0:03.09 httpd
24668 nobody 25 0 47496 13m 3868 R 3.3 0.2 0:04.69 httpd
24669 nobody 25 0 53856 27m 11m R 3.3 0.3 0:05.04 httpd
24670 nobody 25 0 48160 17m 7056 R 3.3 0.2 0:03.11 httpd
24682 nobody 21 0 47496 13m 3884 R 3.3 0.2 0:04.00 httpd
24690 nobody 22 0 47496 13m 3868 R 3.3 0.2 0:04.02 httpd
24705 nobody 21 0 47496 13m 3868 R 3.3 0.2 0:03.71 httpd
24706 nobody 22 0 47496 13m 3884 R 3.3 0.2 0:04.30 httpd
24710 nobody 24 0 47496 13m 3788 R 3.3 0.2 0:01.09 httpd
24716 nobody 25 0 47496 13m 3868 R 3.3 0.2 0:01.14 httpd
24718 nobody 25 0 47496 13m 3788 R 3.3 0.2 0:02.41 httpd
24719 nobody 22 0 47496 13m 3884 R 3.3 0.2 0:04.30 httpd
24720 nobody 25 0 47496 13m 3788 R 3.3 0.2 0:02.40 httpd

 

Iam sure its not there are not too many attackers ips. Hostgator already checked that. Mysq database is located on separate server, Apache has been optimized by support.

 

Any ideas how to optimize server to work with huge traffic maybe?

 

I still see over 5gb of memory available, you should take a look at your apache optimization again. Is there a control panel on the server? Do the error logs state anything out of the ordinary?

 

Thats the thing there are no important errors in error_log. Can you tell me exactly what can be changed to optimize apache to work better?

All i have done is removing rLimits and increase MaxClients in /usr/local/apache/conf/includes/pre_main_global.conf

 

are you running MPM Prefork? maybe worker is a better option for you

 

check the apache error log. For reduce the load try to use ngnix instead of apache.

 

well how much traffic do u recieve at 1 time??

 

No Iam not running MPM Prefork. You see the problem is in something what conflicts with configuration. Website is working for period of time and then boom server load goes right away up to huge number and website stops responding.

 

Well this website used to work fine with even more traffic. There is something wich conflicts with configurations.

 

Its not traffic problems.

 

Is this not something Hostgator tech support can look at?

If it is not a problem with your website then I would let them fix it - or just move hosts/servers.

 

Yes, iam in touch with Level II ssysadmins at Hostgator. They are trying to fix this problem, but doesnt look like they will fix it.

What host can you suggest similar to hostgator? Iam running 8Gb memory server with them.

 

Hi hakzona,

I have read several stories about people finding "WP Super Cache" plugin to be very helpful in reducing server load.

Give that a try if you have not.

 

I use RapidSwitch in the UK for dedicated servers - excellent so far.

 

Why not get them to reinstall Apache, it's not a huge job and may just fix everything (?)

 

From the sounds of it, you are using cPanel. There is a large discussion going on about the way that Apache is restarted due to log rotation in the current released build. I would browse through this thread and see if some of the tips posted will help:

http://forums.cpanel.net/f5/11-25-current-apache-restarting-every-hour-134281.html

 

It could be b'coz some plugin you have installed. Same thing happend to me, i installed some plugin which used to overload my server. I disabled it. Everything went back to normal then. Well that was my solution, i'm not sure about, what realy causing yours.