too many httpd processes, requiring restart/reboot of server

All,

I'm having a problem with my dedicated web sever (3.0 GHZ P4 Processor with 2 GB of RAM). It seems as though, during a certain time of day, my server turns into a vegitative state because there are too many httpd processes running at once. I'm unsure if this is due to my mysql database or if there are just too many requests coming at once that my server can't handle. I constantly need to restart apache in order to "revive" my web server from this state.

I've been going through countless forums to find answers to tweak my apache server variables, along with my mysql variables, and I can't seem to find a resolution to my problem.

I run a site similar to statcounter and sitemeter, where subscribers of my tool insert a small script into there blog or site, and my site tracks and analyzes their visitor statistics for them. I have around 60,000 subscribers

Here's a brief background of my site:


1) Received close to 2 million hits per day (accoding to webalizer), before the problems started occuring, now it's down to 900K per day.

2) The peak times during my site are 9AM - 12PM (rangning from 47K hits per hour to 54K hits) and 9PM - 1 AM (same range). I believe the morning traffic comes heavily from Hong Kong, and the night traffic comes mainly from the US.

3) The oddity in my problem is that, my server only spawns these massive httpd processes during the morning traffic (9AM-12PM) but not during the night traffic (9pm-1am), even though during these time periods, the hit traffic is almost the same in number.


Here's all relevant Apache Configuration:

KeepAlive Off

<IfModule prefork.c>
StartServers 15
MinSpareServers 8
MaxSpareServers 25
MaxClients 150
MaxRequestsPerChild 10000
</IfModule>


<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 10000
</IfModule>


Here's all relevant mysql information:

- 1 database, with 7.12 GB of data
- 700+ tables

SHOW VARIABLES:

| back_log | 50 |
| basedir | /usr/ |
| bdb_cache_size | 8388600 |
| bdb_log_buffer_size | 8388608 |
| bdb_home | /var/lib/mysql/ |
| bdb_max_lock | 10000 |
| bdb_logdir | |
| bdb_shared_data | OFF |
| bdb_tmpdir | /tmp/
| binlog_cache_size | 32768 |
| character_set | latin1 |

| concurrent_insert | ON |
| connect_timeout | 5 |
| datadir | /var/lib/mysql/ |
| delay_key_write | ON |
| delayed_insert_limit | 100 |
| delayed_insert_timeout | 300 |
| delayed_queue_size | 1000 |
| flush | OFF |
| flush_time | 0 |
| have_bdb | YES |
| have_gemini | NO |
| have_innodb | YES |
| have_isam | YES |
| have_raid | NO |
| have_openssl | NO |
| init_file | |
| innodb_additional_mem_pool_size | 12249088 |
| innodb_buffer_pool_size | 489684992 | |
| innodb_data_home_dir | |
| innodb_file_io_threads | 4 |
| innodb_force_recovery | 0 |
| innodb_thread_concurrency | 8 |
| innodb_flush_log_at_trx_commit | 1 |
| innodb_fast_shutdown | ON |
| innodb_flush_method | |
| innodb_lock_wait_timeout | 50 |
| innodb_log_arch_dir | |
| innodb_log_archive | OFF |
| innodb_log_buffer_size | 1048576 |
| innodb_log_file_size | 5242880 |
| innodb_log_files_in_group | 2 | |
| innodb_mirrored_log_groups | 1 |
| interactive_timeout | 28800 |
| join_buffer_size | 131072 |
| key_buffer_size | 1202647040 |
| large_files_support | ON |
| locked_in_memory | OFF |
| log | OFF |
| log_update | OFF |
| log_bin | OFF |
| log_slave_updates | OFF |
| log_long_queries | ON |
| long_query_time | 10 |
| low_priority_updates | OFF |
| lower_case_table_names | 0 |
| max_allowed_packet | 1048576 |
| max_binlog_cache_size | 4294967295 |
| max_binlog_size | 1073741824 |
| max_connections | 16384 |
| max_connect_errors | 10 |
| max_delayed_threads | 20 |
| max_heap_table_size | 16777216 |
| max_join_size | 4294967295 |
| max_sort_length | 1024 |
| max_user_connections | 16000 |
| max_tmp_tables | 32 |
| max_write_lock_count | 4294967295 |
| myisam_max_extra_sort_file_size | 256 |
| myisam_max_sort_file_size | 2047 |
| myisam_recover_options | 0 |
| myisam_sort_buffer_size | 8388608 |
| net_buffer_length | 16384 |
| net_read_timeout | 30 |
| net_retry_count | 10 |
| net_write_timeout | 60 |
| open_files_limit | 0 | |
| port | 3306 |
| protocol_version | 10 |
| record_buffer | 131072 |
| record_rnd_buffer | 1998848 |
| query_buffer_size | 0 |
| safe_show_database | ON |
| server_id | 0 |

| slave_net_timeout | 3600 |
| skip_locking | ON |
| skip_networking | OFF |
| skip_show_database | OFF |
| slow_launch_time | 2 | |
| sort_buffer | 6999992 |
| sql_mode | 0 |
| table_cache | 16384 |
| table_type | MYISAM |
| thread_cache_size | 0 |
| thread_stack | 65536 |
| transaction_isolation | READ-COMMITTED |
| timezone | EST |
| tmp_table_size | 33554432 |
| version | 3.23.58-log |
| wait_timeout | 28800



I have IonCube PHP Accelerator installed my server as well, which I read would optimize my php scripts. I believe this did speed up the compilation of my scripts but this didn't resolve the problem I have of my server spawning too many httpd processes.

I'm a web programmer and have no prior experience with maintaning and configuring web servers.

I would appreciate anyones help with my problem.

 

I would proberly raise the startservers variable in apache so it doesn't have to spend too much time starting new servers. Does apache complain about anything in the error log? (with maxclients 150 I'm wondering if you're hitting that limit).

Could you also post the output of these:

mysqladmin extended-status?
ps auxw (when it's loaded)?

In my experience eAccelerator is the best php accelerator around - however I've never tried IonCube.

 

I thought ioncube and eAccelerator were pretty much the same? based on the same caching framework?

Oddly enough, I was playing around with some variables last night after I posted this entry and my server has been fine so far. I suspect it won't be long till my server starts this problem again, I'l lbe sure to post up the informatoin requested once it does.

What do you think is a good number for StartServers to be?

 

The start servers is not _so_ important in apache2 (in apache 1.x a wrong value could really mess up things) but if you see what the minimum number of servers you seem to have running (check with ps auxw |grep apache | wc -l) set it to something close to that. If you enable the server-status module in apache (config file should have an example) you can see what your apache processes are doing, like idle, starting, waiting for keepalive etc.

With 2 million hits per day I'd proberly consider a second webserver too.

 

I had the exact same problems with one of my sites, I was forced to get a second dedicated box to run the sql off of.

 

1 webserver for apache and another dedicated box for mysql?

 

It's loaded right now, heres the output of mysqladmin

+--------------------------+----------+
| Variable_name | Value |
+--------------------------+----------+
| Aborted_clients | 23 |
| Aborted_connects | 3 |
| Bytes_received | 19567771 |
| Bytes_sent | 35032771 |
| Com_admin_commands | 0 |
| Com_alter_table | 0 |
| Com_analyze | 0 |
| Com_backup_table | 0 |
| Com_begin | 0 |
| Com_change_db | 16509 |
| Com_change_master | 0 |
| Com_check | 0 |
| Com_commit | 0 |
| Com_create_db | 0 |
| Com_create_function | 0 |
| Com_create_index | 0 |
| Com_create_table | 0 |
| Com_delete | 2714 |
| Com_drop_db | 0 |
| Com_drop_function | 0 |
| Com_drop_index | 0 |
| Com_drop_table | 0 |
| Com_flush | 0 |
| Com_grant | 0 |
| Com_insert | 25988 |
| Com_insert_select | 0 |
| Com_kill | 0 |
| Com_load | 0 |
| Com_load_master_table | 0 |
| Com_lock_tables | 0 |
| Com_optimize | 0 |
| Com_purge | 0 |
| Com_rename_table | 0 |
| Com_repair | 0 |
| Com_replace | 1 |
| Com_replace_select | 0 |
| Com_reset | 0 |
| Com_restore_table | 0 |
| Com_revoke | 0 |
| Com_rollback | 0 |
| Com_select | 89575 |
| Com_set_option | 277 |
| Com_show_binlogs | 0 |
| Com_show_create | 0 |
| Com_show_databases | 0 |
| Com_show_fields | 0 |
| Com_show_grants | 0 |
| Com_show_keys | 0 |
| Com_show_logs | 0 |
| Com_show_master_status | 0 |
| Com_show_open_tables | 0 |
| Com_show_processlist | 0 |
| Com_show_slave_status | 0 |
| Com_show_status | 1 |
| Com_show_innodb_status | 0 |
| Com_show_tables | 0 |
| Com_show_variables | 0 |
| Com_slave_start | 0 |
| Com_slave_stop | 0 |
| Com_truncate | 0 |
| Com_unlock_tables | 0 |
| Com_update | 45028 |
| Connections | 16777 |
| Created_tmp_disk_tables | 776 |
| Created_tmp_tables | 7612 |
| Created_tmp_files | 0 |
| Delayed_insert_threads | 0 |
| Delayed_writes | 0 |
| Delayed_errors | 0 |
| Flush_commands | 1 |
| Handler_delete | 2708 |
| Handler_read_first | 91 |
| Handler_read_key | 3948173 |
| Handler_read_next | 97411893 |
| Handler_read_prev | 0 |
| Handler_read_rnd | 303859 |
| Handler_read_rnd_next | 2990629 |
| Handler_update | 3335123 |
| Handler_write | 631217 |
| Key_blocks_used | 21720 |
| Key_read_requests | 3991493 |
| Key_reads | 20496 |
| Key_write_requests | 64905 |
| Key_writes | 58766 |
| Max_used_connections | 20 |
| Not_flushed_key_blocks | 0 |
| Not_flushed_delayed_rows | 0 |
| Open_tables | 589 |
| Open_files | 1089 |
| Open_streams | 0 |
| Opened_tables | 595 |
| Questions | 199705 |
| Select_full_join | 0 |
| Select_full_range_join | 0 |
| Select_range | 0 |
| Select_range_check | 0 |
| Select_scan | 84 |
| Slave_running | OFF |
| Slave_open_temp_tables | 0 |
| Slow_launch_threads | 0 |
| Slow_queries | 8 |
| Sort_merge_passes | 0 |
| Sort_range | 9115 |
| Sort_rows | 252313 |
| Sort_scan | 6253 |
| Table_locks_immediate | 162948 |
| Table_locks_waited | 359 |
| Threads_cached | 0 |
| Threads_created | 16776 |
| Threads_connected | 4 |
| Threads_running | 1 |
| Uptime | 3805 |
+--------------------------+----------+



here it is for ps aux


USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 3000 460 ? S 10:03 0:00 init [3]
root 2 0.0 0.0 0 0 ? SW 10:03 0:00 [migration/0]
root 3 0.0 0.0 0 0 ? SWN 10:03 0:00 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? SW 10:03 0:00 [migration/1]
root 5 0.0 0.0 0 0 ? SWN 10:03 0:00 [ksoftirqd/1]
root 6 0.0 0.0 0 0 ? SW< 10:03 0:00 [events/0]
root 7 0.0 0.0 0 0 ? SW< 10:03 0:00 [events/1]
root 8 0.0 0.0 0 0 ? SW< 10:03 0:00 [khelper]
root 21 0.0 0.0 0 0 ? SW< 10:03 0:00 [kacpid]
root 101 0.3 0.0 0 0 ? SW< 10:03 0:04 [kblockd/0]
root 102 0.0 0.0 0 0 ? SW< 10:03 0:00 [kblockd/1]
root 110 0.0 0.0 0 0 ? SW 10:03 0:00 [khubd]
root 168 0.0 0.0 0 0 ? SW 10:03 0:00 [pdflush]
root 169 0.0 0.0 0 0 ? SW 10:03 0:00 [pdflush]
root 171 0.0 0.0 0 0 ? SW< 10:03 0:00 [aio/0]
root 170 0.0 0.0 0 0 ? SW 10:03 0:00 [kswapd0]
root 172 0.0 0.0 0 0 ? SW< 10:03 0:00 [aio/1]
root 265 0.0 0.0 0 0 ? SW 10:03 0:00 [kseriod]
root 452 0.0 0.0 0 0 ? SW< 10:03 0:00 [ata/0]
root 453 0.0 0.0 0 0 ? SW< 10:03 0:00 [ata/1]
root 458 0.0 0.0 0 0 ? SW 10:03 0:00 [scsi_eh_0]
root 459 0.0 0.0 0 0 ? SW 10:03 0:00 [scsi_eh_1]
root 474 0.0 0.0 0 0 ? SW 10:03 0:00 [kjournald]
root 1566 0.0 0.0 0 0 ? SW 10:03 0:00 [kjournald]
root 2004 0.0 0.0 1864 568 ? S 10:03 0:00 syslogd -m 0
root 2008 0.0 0.0 2388 448 ? S 10:03 0:00 klogd -x
root 2018 0.0 0.0 1872 456 ? S 10:03 0:00 irqbalance
root 2069 0.0 0.0 3084 464 ? S 10:03 0:00 /usr/sbin/acpid
named 2079 0.0 0.1 36572 2764 ? S 10:03 0:00 /usr/sbin/named -u named -c /etc/named.conf -u named -t /var/named/ru
root 2135 0.0 0.0 4940 1424 ? S 10:03 0:00 /usr/sbin/sshd
root 2148 0.0 0.0 2840 856 ? S 10:03 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root 2161 0.0 0.0 4532 1152 ? S 10:03 0:00 /bin/sh /usr/bin/safe_mysqld --defaults-file=/etc/my.cnf --pid-file=/
mysql 2190 16.2 27.2 1834300 562636 ? S 10:03 3:18 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --data
postgres 2245 0.0 0.0 19616 1996 ? S 10:03 0:00 /usr/bin/postmaster -p 5432 -D /var/lib/pgsql/data
postgres 2247 0.0 0.0 10416 1796 ? S 10:03 0:00 postgres: stats buffer process
postgres 2248 0.0 0.0 9424 1840 ? S 10:03 0:00 postgres: stats collector process
ntp 2261 0.0 0.2 5408 5408 ? SL 10:03 0:00 ntpd -U ntp -p /var/run/ntpd.pid -g
root 2276 0.0 0.0 4112 872 ? S 10:03 0:00 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -stderrlogger=/u
root 2278 0.0 0.0 4780 432 ? S 10:03 0:00 /usr/lib/courier-imap/sbin/courierlogger imapd
root 2288 0.0 0.0 4952 872 ? S 10:03 0:00 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -stderrlogger=/u
root 2290 0.0 0.0 3636 436 ? S 10:03 0:00 /usr/lib/courier-imap/sbin/courierlogger imapd-ssl
root 2298 0.0 0.0 5144 880 ? S 10:03 0:00 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -stderrlogger=/u
root 2300 0.0 0.0 4280 568 ? S 10:03 0:00 /usr/lib/courier-imap/sbin/courierlogger pop3d
root 2309 0.0 0.0 4600 876 ? S 10:03 0:00 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -stderrlogger=/u
root 2311 0.0 0.0 3728 432 ? S 10:03 0:00 /usr/lib/courier-imap/sbin/courierlogger pop3d-ssl
qmails 2320 0.0 0.0 2752 356 ? S 10:03 0:00 qmail-send
qmaill 2324 0.0 0.0 3336 412 ? S 10:03 0:00 splogger qmail
root 2325 0.0 0.0 2972 324 ? S 10:03 0:00 qmail-lspawn ./Maildir/
qmailr 2326 0.0 0.0 1952 344 ? S 10:03 0:00 qmail-rspawn
qmailq 2327 0.0 0.0 2344 300 ? S 10:03 0:00 qmail-clean
popuser 2350 0.0 1.0 25772 20768 ? S 10:03 0:00 /usr/bin/spamd --username=popuser --daemonize --helper-home-dir=/var/
popuser 2352 0.0 1.0 26788 20768 ? S 10:03 0:00 /usr/bin/spamd --username=popuser --daemonize --helper-home-dir=/var/
root 2473 0.0 0.2 14856 5308 ? S 10:03 0:00 /usr/local/psa/admin/bin/httpsd
psaadm 2483 0.0 0.2 14856 5312 ? S 10:03 0:00 /usr/local/psa/admin/bin/httpsd
psaadm 2484 0.0 0.2 14856 5312 ? S 10:03 0:00 /usr/local/psa/admin/bin/httpsd
psaadm 2485 0.0 0.2 14856 5312 ? S 10:03 0:00 /usr/local/psa/admin/bin/httpsd
psaadm 2486 0.0 0.2 14856 5312 ? S 10:03 0:00 /usr/local/psa/admin/bin/httpsd
psaadm 2487 0.0 0.2 14856 5312 ? S 10:03 0:00 /usr/local/psa/admin/bin/httpsd
root 2580 0.0 0.0 1968 644 ? S 10:03 0:00 crond
daemon 2631 0.0 0.0 2132 600 ? S 10:03 0:00 /usr/sbin/atd
dbus 2641 0.0 0.0 3600 808 ? S 10:03 0:00 dbus-daemon-1 --system
root 2652 0.0 0.0 2620 340 tty1 S 10:03 0:00 /sbin/mingetty tty1
root 2653 0.0 0.0 2312 344 tty2 S 10:03 0:00 /sbin/mingetty tty2
root 2654 0.0 0.0 3032 344 tty3 S 10:03 0:00 /sbin/mingetty tty3
root 2655 0.0 0.0 3240 344 tty4 S 10:03 0:00 /sbin/mingetty tty4
root 2656 0.0 0.0 2056 344 tty5 S 10:03 0:00 /sbin/mingetty tty5
root 2657 0.0 0.0 2656 344 tty6 S 10:03 0:00 /sbin/mingetty tty6
root 4361 0.0 0.2 5408 5408 ? S 10:09 0:00 ntpd -U ntp -p
root 4449 0.0 0.0 5448 1064 pts/0 S 10:09 0:00 su root
root 4463 0.0 0.0 6160 1392 pts/0 S 10:09 0:00 bash
root 7685 0.1 0.5 26396 11268 ? S 10:20 0:00 /usr/sbin/httpd
root 7693 0.0 0.5 26340 11204 ? S 10:20 0:00 /usr/sbin/httpd
apache 7694 0.1 0.5 26872 12220 ? S 10:20 0:00 /usr/sbin/httpd
apache 7695 0.0 0.5 26892 12196 ? S 10:20 0:00 /usr/sbin/httpd
apache 7696 0.0 0.5 26900 12280 ? S 10:20 0:00 /usr/sbin/httpd
apache 7697 0.0 0.5 26864 12240 ? S 10:20 0:00 /usr/sbin/httpd
apache 7698 0.0 0.5 26864 12172 ? S 10:20 0:00 /usr/sbin/httpd
apache 7704 0.0 0.5 26788 12072 ? S 10:20 0:00 /usr/sbin/httpd
apache 7705 24.3 0.5 26680 11812 ? R 10:20 0:47 /usr/sbin/httpd
apache 7707 0.1 0.5 26864 12200 ? S 10:20 0:00 /usr/sbin/httpd
apache 7708 0.1 0.5 26872 12216 ? S 10:20 0:00 /usr/sbin/httpd
apache 7709 0.1 0.5 26864 12212 ? S 10:20 0:00 /usr/sbin/httpd
apache 7710 0.0 0.5 26884 12252 ? S 10:20 0:00 /usr/sbin/httpd
apache 7715 10.4 0.5 26824 12180 ? R 10:20 0:20 /usr/sbin/httpd
apache 7717 0.1 0.5 26864 12200 ? S 10:20 0:00 /usr/sbin/httpd
apache 7718 0.0 0.5 26840 12216 ? S 10:20 0:00 /usr/sbin/httpd
apache 7721 0.1 0.5 26852 12200 ? S 10:20 0:00 /usr/sbin/httpd
apache 7723 0.1 0.5 26856 12208 ? S 10:20 0:00 /usr/sbin/httpd
apache 7724 0.0 0.5 26888 12240 ? S 10:20 0:00 /usr/sbin/httpd
apache 7727 0.1 0.5 26856 12196 ? S 10:20 0:00 /usr/sbin/httpd
apache 7728 0.1 0.5 26864 12192 ? S 10:20 0:00 /usr/sbin/httpd
apache 7729 0.0 0.5 26884 12228 ? S 10:20 0:00 /usr/sbin/httpd
apache 7730 0.1 0.5 26868 12176 ? S 10:20 0:00 /usr/sbin/httpd
apache 7732 0.0 0.5 26868 12176 ? S 10:20 0:00 /usr/sbin/httpd
apache 7734 4.1 0.5 26884 12228 ? R 10:20 0:08 /usr/sbin/httpd
apache 7735 0.0 0.5 26864 12176 ? S 10:20 0:00 /usr/sbin/httpd
apache 7736 0.1 0.5 26852 12200 ? S 10:20 0:00 /usr/sbin/httpd
apache 7737 0.1 0.5 26856 12192 ? S 10:20 0:00 /usr/sbin/httpd
apache 7740 0.0 0.5 26892 12248 ? S 10:20 0:00 /usr/sbin/httpd
apache 7741 0.0 0.5 26860 12224 ? S 10:20 0:00 /usr/sbin/httpd
apache 7743 0.0 0.5 26872 12212 ? S 10:20 0:00 /usr/sbin/httpd
apache 7744 0.0 0.5 26872 12176 ? S 10:20 0:00 /usr/sbin/httpd
apache 7748 0.1 0.5 26876 12212 ? S 10:20 0:00 /usr/sbin/httpd
apache 7749 0.0 0.5 26876 12176 ? S 10:20 0:00 /usr/sbin/httpd
apache 7751 0.0 0.5 26888 12244 ? S 10:20 0:00 /usr/sbin/httpd
apache 7752 15.2 0.5 26680 11892 ? R 10:20 0:29 /usr/sbin/httpd
apache 7753 0.0 0.5 26816 12192 ? S 10:20 0:00 /usr/sbin/httpd
apache 7758 13.2 0.5 26908 12200 ? R 10:20 0:25 /usr/sbin/httpd
apache 7759 0.0 0.5 26892 12192 ? S 10:20 0:00 /usr/sbin/httpd
apache 7760 0.1 0.5 26864 12212 ? S 10:20 0:00 /usr/sbin/httpd
apache 7763 0.0 0.5 26532 11656 ? S 10:20 0:00 /usr/sbin/httpd
apache 7766 0.1 0.5 26856 12208 ? S 10:20 0:00 /usr/sbin/httpd
apache 7767 0.1 0.5 26868 12220 ? S 10:20 0:00 /usr/sbin/httpd
apache 7770 0.0 0.5 26884 12252 ? S 10:20 0:00 /usr/sbin/httpd
apache 7771 0.0 0.5 26864 12192 ? S 10:20 0:00 /usr/sbin/httpd
apache 7772 0.0 0.5 26888 12276 ? S 10:20 0:00 /usr/sbin/httpd
apache 7773 0.0 0.5 26892 12196 ? S 10:20 0:00 /usr/sbin/httpd
apache 7774 1.2 0.5 26892 12240 ? R 10:20 0:02 /usr/sbin/httpd
apache 7775 0.0 0.5 26864 12188 ? S 10:20 0:00 /usr/sbin/httpd
apache 7777 0.9 0.5 26896 12260 ? R 10:20 0:01 /usr/sbin/httpd
apache 7779 0.1 0.5 26876 12188 ? S 10:20 0:00 /usr/sbin/httpd
apache 7781 0.1 0.5 26864 12208 ? S 10:20 0:00 /usr/sbin/httpd
apache 7783 20.4 0.5 26832 12208 ? R 10:20 0:39 /usr/sbin/httpd
apache 7784 0.1 0.5 26892 12244 ? S 10:20 0:00 /usr/sbin/httpd
apache 7787 0.1 0.5 26888 12236 ? S 10:20 0:00 /usr/sbin/httpd
apache 7792 0.0 0.5 26872 12220 ? S 10:20 0:00 /usr/sbin/httpd
apache 7794 0.0 0.5 27000 12312 ? S 10:20 0:00 /usr/sbin/httpd
apache 7795 0.0 0.5 26880 12228 ? S 10:20 0:00 /usr/sbin/httpd
apache 7797 0.0 0.5 26892 12232 ? S 10:20 0:00 /usr/sbin/httpd
apache 7798 0.0 0.5 26888 12236 ? S 10:20 0:00 /usr/sbin/httpd
apache 7799 0.1 0.5 26888 12220 ? S 10:20 0:00 /usr/sbin/httpd
apache 7800 0.0 0.5 26864 12184 ? S 10:20 0:00 /usr/sbin/httpd







I'm not exactly sure what's going on but yesterday, my site was running totally fine. No 20 httpd processes running at once, and my site was running smoothly.

But just as I dreaded, the site is back to where it was before yesterday. I'm at work at the moment, and I constantly need to restart apache to get my server running again. I've looking at the processes running via top, and I noticed some httpd processes are chewing up a hell of alot of CPU:

check it out:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4386 apache 25 0 27496 11m 7916 R 83.0 0.6 0:52.79 httpd
4362 apache 25 0 27536 12m 7920 R 49.8 0.6 0:20.66 httpd
4365 apache 25 0 27496 11m 7900 R 49.8 0.6 0:29.91 httpd
2193 mysql 15 0 1779m 538m 3084 S 11.3 26.7 1:07.20 mysqld
4323 apache 15 0 27476 11m 7860 S 1.0 0.6 0:00.35 httpd
4318 apache 15 0 27480 11m 7868 S 0.7 0.6 0:00.22 httpd
4312 apache 15 0 27476 11m 7860 S 0.3 0.6 0:00.28 httpd
4316 apache 15 0 27488 11m 7868 S 0.3 0.6 0:00.24 httpd
4319 apache 15 0 27476 11m 7864 S 0.3 0.6 0:00.29 httpd
4355 apache 15 0 27500 11m 7872 S 0.3 0.6 0:00.27 httpd
4361 apache 15 0 27464 11m 7868 S 0.3 0.6 0:00.33 httpd
4366 apache 15 0 27480 11m 7872 S 0.3 0.6 0:00.29 httpd
4367 apache 15 0 27476 11m 7868 S 0.3 0.6 0:00.28 httpd
4368 apache 15 0 27476 11m 7860 S 0.3 0.6 0:00.32 httpd
4373 apache 15 0 27476 11m 7868 S 0.3 0.6 0:00.31 httpd
4375 apache 15 0 27480 11m 7864 S 0.3 0.6 0:00.22 httpd
4376 apache 16 0 27488 11m 7860 S 0.3 0.6 0:00.27 httpd
4378 apache 16 0 27476 11m 7864 S 0.3 0.6 0:00.24 httpd
4385 apache 15 0 27476 11m 7860 S 0.3 0.6 0:00.29 httpd
4795 apache 16 0 27508 11m 7868 S 0.3 0.6 0:00.16 httpd
4817 root 16 0 2188 924 724 R 0.3 0.0 0:00.28 top
4889 apache 16 0 27472 11m 7856 S 0.3 0.6 0:00.14 httpd
4895 apache 15 0 27436 11m 7860 S 0.3 0.6 0:00.12 httpd



I've checked my error_logs, and no errors are occurring out of the ordinary. Does any one have a clue about whats going on?

IonCube, I appreciate your replies. Yes, all the customers do access their stats via the same server, that records the stats. The point I can't get passed is that, why it's happening in the morning and not in the night time? The night time experiences the same amount of traffic as the morning, why would it occur in the morning and not during the night? The server and the mysql config is set up exactly the same at both times, if it's running smoothly at night when the server experiences the same amoutn of traffic, why is dying out in the morning? Would anyone suspect foul play? If so, how can I tell?

One other thing is that, the page that contains the stats for my users is the main one that's causing the hold up i believe. If you navigate through my site, all the other pages seem to load up lightning fast, but when you do try to access the page that contains the user stats, that's when slow down begins. Obviously, this has some connection with the mysql db, but what could it be? I'm pretty sure I have my indices on the correct columns and also, I have a cron job set up to do a daily optimize of the tables each night.

 

One thing that springs to mind is that you have lots of mysql threads created and none cached, try upping the value for thread_cache in my.cnf

I've read a suggestion of table_cache = opened table / max_used_connection which would make it 29 for you.

Also this one:

Created_tmp_disk_tables | 776 |

I suggest you try and raise tmp_table_size to try and keep the tmp tables in memory. It's not an alarming number compated to the created_tmp_tables but I'd try and up it a bit to see what happens.

 

blinxdk, I've made the updates you requested.

set-variable = thread_cache_size = 29
set-variable = tmp_table_size = 103554432
set-variable = table_cache = 6090004 (doubled the table_cache)

Let's how tomorrow turns out, I appreciate your help! thanks.

 

zonequestz,

I am having the same issue right now and I am restarting the Apache over and over.

did you set up the start server parameter?

did this issue was resolved for you?

really appreciate your response.

 

You could always go to a managed server (where they look after it for you), can recommend a good one (as i work for them )

 

I've had a very similar problem to the one you mention and the cause was excessively long MySQL queries.

Each Apache process opens up a connection to the database server and then waits 30 or 40 seconds to get it's result back. As you can imagine, at that rate, Apache hits MaxClients very quickly and the site appears to go slow for some people and timeout for others. You can gain some extra time by raising MaxClients but you won't gain very much extra time that way.

The solution for us was to look through the MySQL slow logs for frequent queries that took a long time (mysqldumpslow is a great tool for analysing those...) and optimise the main culprits. (Then we went through the subversion blame logs to find out who wrote all the dodgy queries )