1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

too many httpd processes, requiring restart/reboot of server

Discussion in 'Apache' started by zonequestz, Feb 26, 2006.

  1. #1
    All,

    I'm having a problem with my dedicated web sever (3.0 GHZ P4 Processor with 2 GB of RAM). It seems as though, during a certain time of day, my server turns into a vegitative state because there are too many httpd processes running at once. I'm unsure if this is due to my mysql database or if there are just too many requests coming at once that my server can't handle. I constantly need to restart apache in order to "revive" my web server from this state.

    I've been going through countless forums to find answers to tweak my apache server variables, along with my mysql variables, and I can't seem to find a resolution to my problem.

    I run a site similar to statcounter and sitemeter, where subscribers of my tool insert a small script into there blog or site, and my site tracks and analyzes their visitor statistics for them. I have around 60,000 subscribers

    Here's a brief background of my site:


    1) Received close to 2 million hits per day (accoding to webalizer), before the problems started occuring, now it's down to 900K per day.

    2) The peak times during my site are 9AM - 12PM (rangning from 47K hits per hour to 54K hits) and 9PM - 1 AM (same range). I believe the morning traffic comes heavily from Hong Kong, and the night traffic comes mainly from the US.

    3) The oddity in my problem is that, my server only spawns these massive httpd processes during the morning traffic (9AM-12PM) but not during the night traffic (9pm-1am), even though during these time periods, the hit traffic is almost the same in number.



    Here's all relevant Apache Configuration:

    KeepAlive Off

    <IfModule prefork.c>
    StartServers 15
    MinSpareServers 8
    MaxSpareServers 25
    MaxClients 150
    MaxRequestsPerChild 10000
    </IfModule>


    <IfModule worker.c>
    StartServers 2
    MaxClients 150
    MinSpareThreads 25
    MaxSpareThreads 75
    ThreadsPerChild 25
    MaxRequestsPerChild 10000
    </IfModule>


    Here's all relevant mysql information:

    - 1 database, with 7.12 GB of data
    - 700+ tables

    SHOW VARIABLES:

    | back_log | 50 |
    | basedir | /usr/ |
    | bdb_cache_size | 8388600 |
    | bdb_log_buffer_size | 8388608 |
    | bdb_home | /var/lib/mysql/ |
    | bdb_max_lock | 10000 |
    | bdb_logdir | |
    | bdb_shared_data | OFF |
    | bdb_tmpdir | /tmp/
    | binlog_cache_size | 32768 |
    | character_set | latin1 |

    | concurrent_insert | ON |
    | connect_timeout | 5 |
    | datadir | /var/lib/mysql/ |
    | delay_key_write | ON |
    | delayed_insert_limit | 100 |
    | delayed_insert_timeout | 300 |
    | delayed_queue_size | 1000 |
    | flush | OFF |
    | flush_time | 0 |
    | have_bdb | YES |
    | have_gemini | NO |
    | have_innodb | YES |
    | have_isam | YES |
    | have_raid | NO |
    | have_openssl | NO |
    | init_file | |
    | innodb_additional_mem_pool_size | 12249088 |
    | innodb_buffer_pool_size | 489684992 | |
    | innodb_data_home_dir | |
    | innodb_file_io_threads | 4 |
    | innodb_force_recovery | 0 |
    | innodb_thread_concurrency | 8 |
    | innodb_flush_log_at_trx_commit | 1 |
    | innodb_fast_shutdown | ON |
    | innodb_flush_method | |
    | innodb_lock_wait_timeout | 50 |
    | innodb_log_arch_dir | |
    | innodb_log_archive | OFF |
    | innodb_log_buffer_size | 1048576 |
    | innodb_log_file_size | 5242880 |
    | innodb_log_files_in_group | 2 | |
    | innodb_mirrored_log_groups | 1 |
    | interactive_timeout | 28800 |
    | join_buffer_size | 131072 |
    | key_buffer_size | 1202647040 |
    | large_files_support | ON |
    | locked_in_memory | OFF |
    | log | OFF |
    | log_update | OFF |
    | log_bin | OFF |
    | log_slave_updates | OFF |
    | log_long_queries | ON |
    | long_query_time | 10 |
    | low_priority_updates | OFF |
    | lower_case_table_names | 0 |
    | max_allowed_packet | 1048576 |
    | max_binlog_cache_size | 4294967295 |
    | max_binlog_size | 1073741824 |
    | max_connections | 16384 |
    | max_connect_errors | 10 |
    | max_delayed_threads | 20 |
    | max_heap_table_size | 16777216 |
    | max_join_size | 4294967295 |
    | max_sort_length | 1024 |
    | max_user_connections | 16000 |
    | max_tmp_tables | 32 |
    | max_write_lock_count | 4294967295 |
    | myisam_max_extra_sort_file_size | 256 |
    | myisam_max_sort_file_size | 2047 |
    | myisam_recover_options | 0 |
    | myisam_sort_buffer_size | 8388608 |
    | net_buffer_length | 16384 |
    | net_read_timeout | 30 |
    | net_retry_count | 10 |
    | net_write_timeout | 60 |
    | open_files_limit | 0 | |
    | port | 3306 |
    | protocol_version | 10 |
    | record_buffer | 131072 |
    | record_rnd_buffer | 1998848 |
    | query_buffer_size | 0 |
    | safe_show_database | ON |
    | server_id | 0 |

    | slave_net_timeout | 3600 |
    | skip_locking | ON |
    | skip_networking | OFF |
    | skip_show_database | OFF |
    | slow_launch_time | 2 | |
    | sort_buffer | 6999992 |
    | sql_mode | 0 |
    | table_cache | 16384 |
    | table_type | MYISAM |
    | thread_cache_size | 0 |
    | thread_stack | 65536 |
    | transaction_isolation | READ-COMMITTED |
    | timezone | EST |
    | tmp_table_size | 33554432 |
    | version | 3.23.58-log |
    | wait_timeout | 28800



    I have IonCube PHP Accelerator installed my server as well, which I read would optimize my php scripts. I believe this did speed up the compilation of my scripts but this didn't resolve the problem I have of my server spawning too many httpd processes.

    I'm a web programmer and have no prior experience with maintaning and configuring web servers.

    I would appreciate anyones help with my problem.
    SEMrush
     
    zonequestz, Feb 26, 2006 IP
    SEMrush
  2. blinxdk

    blinxdk Peon

    Messages:
    660
    Likes Received:
    27
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I would proberly raise the startservers variable in apache so it doesn't have to spend too much time starting new servers. Does apache complain about anything in the error log? (with maxclients 150 I'm wondering if you're hitting that limit).

    Could you also post the output of these:

    mysqladmin extended-status?
    ps auxw (when it's loaded)?

    In my experience eAccelerator is the best php accelerator around - however I've never tried IonCube.
     
    blinxdk, Feb 27, 2006 IP
  3. zonequestz

    zonequestz Guest

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I thought ioncube and eAccelerator were pretty much the same? based on the same caching framework?

    Oddly enough, I was playing around with some variables last night after I posted this entry and my server has been fine so far. I suspect it won't be long till my server starts this problem again, I'l lbe sure to post up the informatoin requested once it does.

    What do you think is a good number for StartServers to be?
     
    zonequestz, Feb 27, 2006 IP
  4. blinxdk

    blinxdk Peon

    Messages:
    660
    Likes Received:
    27
    Best Answers:
    0
    Trophy Points:
    0
    #4
    The start servers is not _so_ important in apache2 (in apache 1.x a wrong value could really mess up things) but if you see what the minimum number of servers you seem to have running (check with ps auxw |grep apache | wc -l) set it to something close to that. If you enable the server-status module in apache (config file should have an example) you can see what your apache processes are doing, like idle, starting, waiting for keepalive etc.

    With 2 million hits per day I'd proberly consider a second webserver too.
     
    blinxdk, Feb 27, 2006 IP
  5. Torrentguy

    Torrentguy Member

    Messages:
    87
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    48
    #5
    I had the exact same problems with one of my sites, I was forced to get a second dedicated box to run the sql off of.
     
    Torrentguy, Feb 27, 2006 IP
  6. zonequestz

    zonequestz Guest

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6


    1 webserver for apache and another dedicated box for mysql?
     
    zonequestz, Feb 28, 2006 IP
  7. zonequestz

    zonequestz Guest

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    It's loaded right now, heres the output of mysqladmin

    +--------------------------+----------+
    | Variable_name | Value |
    +--------------------------+----------+
    | Aborted_clients | 23 |
    | Aborted_connects | 3 |
    | Bytes_received | 19567771 |
    | Bytes_sent | 35032771 |
    | Com_admin_commands | 0 |
    | Com_alter_table | 0 |
    | Com_analyze | 0 |
    | Com_backup_table | 0 |
    | Com_begin | 0 |
    | Com_change_db | 16509 |
    | Com_change_master | 0 |
    | Com_check | 0 |
    | Com_commit | 0 |
    | Com_create_db | 0 |
    | Com_create_function | 0 |
    | Com_create_index | 0 |
    | Com_create_table | 0 |
    | Com_delete | 2714 |
    | Com_drop_db | 0 |
    | Com_drop_function | 0 |
    | Com_drop_index | 0 |
    | Com_drop_table | 0 |
    | Com_flush | 0 |
    | Com_grant | 0 |
    | Com_insert | 25988 |
    | Com_insert_select | 0 |
    | Com_kill | 0 |
    | Com_load | 0 |
    | Com_load_master_table | 0 |
    | Com_lock_tables | 0 |
    | Com_optimize | 0 |
    | Com_purge | 0 |
    | Com_rename_table | 0 |
    | Com_repair | 0 |
    | Com_replace | 1 |
    | Com_replace_select | 0 |
    | Com_reset | 0 |
    | Com_restore_table | 0 |
    | Com_revoke | 0 |
    | Com_rollback | 0 |
    | Com_select | 89575 |
    | Com_set_option | 277 |
    | Com_show_binlogs | 0 |
    | Com_show_create | 0 |
    | Com_show_databases | 0 |
    | Com_show_fields | 0 |
    | Com_show_grants | 0 |
    | Com_show_keys | 0 |
    | Com_show_logs | 0 |
    | Com_show_master_status | 0 |
    | Com_show_open_tables | 0 |
    | Com_show_processlist | 0 |
    | Com_show_slave_status | 0 |
    | Com_show_status | 1 |
    | Com_show_innodb_status | 0 |
    | Com_show_tables | 0 |
    | Com_show_variables | 0 |
    | Com_slave_start | 0 |
    | Com_slave_stop | 0 |
    | Com_truncate | 0 |
    | Com_unlock_tables | 0 |
    | Com_update | 45028 |
    | Connections | 16777 |
    | Created_tmp_disk_tables | 776 |
    | Created_tmp_tables | 7612 |
    | Created_tmp_files | 0 |
    | Delayed_insert_threads | 0 |
    | Delayed_writes | 0 |
    | Delayed_errors | 0 |
    | Flush_commands | 1 |
    | Handler_delete | 2708 |
    | Handler_read_first | 91 |
    | Handler_read_key | 3948173 |
    | Handler_read_next | 97411893 |
    | Handler_read_prev | 0 |
    | Handler_read_rnd | 303859 |
    | Handler_read_rnd_next | 2990629 |
    | Handler_update | 3335123 |
    | Handler_write | 631217 |
    | Key_blocks_used | 21720 |
    | Key_read_requests | 3991493 |
    | Key_reads | 20496 |
    | Key_write_requests | 64905 |
    | Key_writes | 58766 |
    | Max_used_connections | 20 |
    | Not_flushed_key_blocks | 0 |
    | Not_flushed_delayed_rows | 0 |
    | Open_tables | 589 |
    | Open_files | 1089 |
    | Open_streams | 0 |
    | Opened_tables | 595 |
    | Questions | 199705 |
    | Select_full_join | 0 |
    | Select_full_range_join | 0 |
    | Select_range | 0 |
    | Select_range_check | 0 |
    | Select_scan | 84 |
    | Slave_running | OFF |
    | Slave_open_temp_tables | 0 |
    | Slow_launch_threads | 0 |
    | Slow_queries | 8 |
    | Sort_merge_passes | 0 |
    | Sort_range | 9115 |
    | Sort_rows | 252313 |
    | Sort_scan | 6253 |
    | Table_locks_immediate | 162948 |
    | Table_locks_waited | 359 |
    | Threads_cached | 0 |
    | Threads_created | 16776 |
    | Threads_connected | 4 |
    | Threads_running | 1 |
    | Uptime | 3805 |
    +--------------------------+----------+



    here it is for ps aux


    USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
    root 1 0.0 0.0 3000 460 ? S 10:03 0:00 init [3]
    root 2 0.0 0.0 0 0 ? SW 10:03 0:00 [migration/0]
    root 3 0.0 0.0 0 0 ? SWN 10:03 0:00 [ksoftirqd/0]
    root 4 0.0 0.0 0 0 ? SW 10:03 0:00 [migration/1]
    root 5 0.0 0.0 0 0 ? SWN 10:03 0:00 [ksoftirqd/1]
    root 6 0.0 0.0 0 0 ? SW< 10:03 0:00 [events/0]
    root 7 0.0 0.0 0 0 ? SW< 10:03 0:00 [events/1]
    root 8 0.0 0.0 0 0 ? SW< 10:03 0:00 [khelper]
    root 21 0.0 0.0 0 0 ? SW< 10:03 0:00 [kacpid]
    root 101 0.3 0.0 0 0 ? SW< 10:03 0:04 [kblockd/0]
    root 102 0.0 0.0 0 0 ? SW< 10:03 0:00 [kblockd/1]
    root 110 0.0 0.0 0 0 ? SW 10:03 0:00 [khubd]
    root 168 0.0 0.0 0 0 ? SW 10:03 0:00 [pdflush]
    root 169 0.0 0.0 0 0 ? SW 10:03 0:00 [pdflush]
    root 171 0.0 0.0 0 0 ? SW< 10:03 0:00 [aio/0]
    root 170 0.0 0.0 0 0 ? SW 10:03 0:00 [kswapd0]
    root 172 0.0 0.0 0 0 ? SW< 10:03 0:00 [aio/1]
    root 265 0.0 0.0 0 0 ? SW 10:03 0:00 [kseriod]
    root 452 0.0 0.0 0 0 ? SW< 10:03 0:00 [ata/0]
    root 453 0.0 0.0 0 0 ? SW< 10:03 0:00 [ata/1]
    root 458 0.0 0.0 0 0 ? SW 10:03 0:00 [scsi_eh_0]
    root 459 0.0 0.0 0 0 ? SW 10:03 0:00 [scsi_eh_1]
    root 474 0.0 0.0 0 0 ? SW 10:03 0:00 [kjournald]
    root 1566 0.0 0.0 0 0 ? SW 10:03 0:00 [kjournald]
    root 2004 0.0 0.0 1864 568 ? S 10:03 0:00 syslogd -m 0
    root 2008 0.0 0.0 2388 448 ? S 10:03 0:00 klogd -x
    root 2018 0.0 0.0 1872 456 ? S 10:03 0:00 irqbalance
    root 2069 0.0 0.0 3084 464 ? S 10:03 0:00 /usr/sbin/acpid
    named 2079 0.0 0.1 36572 2764 ? S 10:03 0:00 /usr/sbin/named -u named -c /etc/named.conf -u named -t /var/named/ru
    root 2135 0.0 0.0 4940 1424 ? S 10:03 0:00 /usr/sbin/sshd
    root 2148 0.0 0.0 2840 856 ? S 10:03 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
    root 2161 0.0 0.0 4532 1152 ? S 10:03 0:00 /bin/sh /usr/bin/safe_mysqld --defaults-file=/etc/my.cnf --pid-file=/
    mysql 2190 16.2 27.2 1834300 562636 ? S 10:03 3:18 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --data
    postgres 2245 0.0 0.0 19616 1996 ? S 10:03 0:00 /usr/bin/postmaster -p 5432 -D /var/lib/pgsql/data
    postgres 2247 0.0 0.0 10416 1796 ? S 10:03 0:00 postgres: stats buffer process
    postgres 2248 0.0 0.0 9424 1840 ? S 10:03 0:00 postgres: stats collector process
    ntp 2261 0.0 0.2 5408 5408 ? SL 10:03 0:00 ntpd -U ntp -p /var/run/ntpd.pid -g
    root 2276 0.0 0.0 4112 872 ? S 10:03 0:00 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -stderrlogger=/u
    root 2278 0.0 0.0 4780 432 ? S 10:03 0:00 /usr/lib/courier-imap/sbin/courierlogger imapd
    root 2288 0.0 0.0 4952 872 ? S 10:03 0:00 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -stderrlogger=/u
    root 2290 0.0 0.0 3636 436 ? S 10:03 0:00 /usr/lib/courier-imap/sbin/courierlogger imapd-ssl
    root 2298 0.0 0.0 5144 880 ? S 10:03 0:00 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -stderrlogger=/u
    root 2300 0.0 0.0 4280 568 ? S 10:03 0:00 /usr/lib/courier-imap/sbin/courierlogger pop3d
    root 2309 0.0 0.0 4600 876 ? S 10:03 0:00 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -stderrlogger=/u
    root 2311 0.0 0.0 3728 432 ? S 10:03 0:00 /usr/lib/courier-imap/sbin/courierlogger pop3d-ssl
    qmails 2320 0.0 0.0 2752 356 ? S 10:03 0:00 qmail-send
    qmaill 2324 0.0 0.0 3336 412 ? S 10:03 0:00 splogger qmail
    root 2325 0.0 0.0 2972 324 ? S 10:03 0:00 qmail-lspawn ./Maildir/
    qmailr 2326 0.0 0.0 1952 344 ? S 10:03 0:00 qmail-rspawn
    qmailq 2327 0.0 0.0 2344 300 ? S 10:03 0:00 qmail-clean
    popuser 2350 0.0 1.0 25772 20768 ? S 10:03 0:00 /usr/bin/spamd --username=popuser --daemonize --helper-home-dir=/var/
    popuser 2352 0.0 1.0 26788 20768 ? S 10:03 0:00 /usr/bin/spamd --username=popuser --daemonize --helper-home-dir=/var/
    root 2473 0.0 0.2 14856 5308 ? S 10:03 0:00 /usr/local/psa/admin/bin/httpsd
    psaadm 2483 0.0 0.2 14856 5312 ? S 10:03 0:00 /usr/local/psa/admin/bin/httpsd
    psaadm 2484 0.0 0.2 14856 5312 ? S 10:03 0:00 /usr/local/psa/admin/bin/httpsd
    psaadm 2485 0.0 0.2 14856 5312 ? S 10:03 0:00 /usr/local/psa/admin/bin/httpsd
    psaadm 2486 0.0 0.2 14856 5312 ? S 10:03 0:00 /usr/local/psa/admin/bin/httpsd
    psaadm 2487 0.0 0.2 14856 5312 ? S 10:03 0:00 /usr/local/psa/admin/bin/httpsd
    root 2580 0.0 0.0 1968 644 ? S 10:03 0:00 crond
    daemon 2631 0.0 0.0 2132 600 ? S 10:03 0:00 /usr/sbin/atd
    dbus 2641 0.0 0.0 3600 808 ? S 10:03 0:00 dbus-daemon-1 --system
    root 2652 0.0 0.0 2620 340 tty1 S 10:03 0:00 /sbin/mingetty tty1
    root 2653 0.0 0.0 2312 344 tty2 S 10:03 0:00 /sbin/mingetty tty2
    root 2654 0.0 0.0 3032 344 tty3 S 10:03 0:00 /sbin/mingetty tty3
    root 2655 0.0 0.0 3240 344 tty4 S 10:03 0:00 /sbin/mingetty tty4
    root 2656 0.0 0.0 2056 344 tty5 S 10:03 0:00 /sbin/mingetty tty5
    root 2657 0.0 0.0 2656 344 tty6 S 10:03 0:00 /sbin/mingetty tty6
    root 4361 0.0 0.2 5408 5408 ? S 10:09 0:00 ntpd -U ntp -p
    root 4449 0.0 0.0 5448 1064 pts/0 S 10:09 0:00 su root
    root 4463 0.0 0.0 6160 1392 pts/0 S 10:09 0:00 bash
    root 7685 0.1 0.5 26396 11268 ? S 10:20 0:00 /usr/sbin/httpd
    root 7693 0.0 0.5 26340 11204 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7694 0.1 0.5 26872 12220 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7695 0.0 0.5 26892 12196 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7696 0.0 0.5 26900 12280 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7697 0.0 0.5 26864 12240 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7698 0.0 0.5 26864 12172 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7704 0.0 0.5 26788 12072 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7705 24.3 0.5 26680 11812 ? R 10:20 0:47 /usr/sbin/httpd
    apache 7707 0.1 0.5 26864 12200 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7708 0.1 0.5 26872 12216 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7709 0.1 0.5 26864 12212 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7710 0.0 0.5 26884 12252 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7715 10.4 0.5 26824 12180 ? R 10:20 0:20 /usr/sbin/httpd
    apache 7717 0.1 0.5 26864 12200 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7718 0.0 0.5 26840 12216 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7721 0.1 0.5 26852 12200 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7723 0.1 0.5 26856 12208 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7724 0.0 0.5 26888 12240 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7727 0.1 0.5 26856 12196 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7728 0.1 0.5 26864 12192 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7729 0.0 0.5 26884 12228 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7730 0.1 0.5 26868 12176 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7732 0.0 0.5 26868 12176 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7734 4.1 0.5 26884 12228 ? R 10:20 0:08 /usr/sbin/httpd
    apache 7735 0.0 0.5 26864 12176 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7736 0.1 0.5 26852 12200 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7737 0.1 0.5 26856 12192 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7740 0.0 0.5 26892 12248 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7741 0.0 0.5 26860 12224 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7743 0.0 0.5 26872 12212 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7744 0.0 0.5 26872 12176 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7748 0.1 0.5 26876 12212 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7749 0.0 0.5 26876 12176 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7751 0.0 0.5 26888 12244 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7752 15.2 0.5 26680 11892 ? R 10:20 0:29 /usr/sbin/httpd
    apache 7753 0.0 0.5 26816 12192 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7758 13.2 0.5 26908 12200 ? R 10:20 0:25 /usr/sbin/httpd
    apache 7759 0.0 0.5 26892 12192 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7760 0.1 0.5 26864 12212 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7763 0.0 0.5 26532 11656 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7766 0.1 0.5 26856 12208 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7767 0.1 0.5 26868 12220 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7770 0.0 0.5 26884 12252 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7771 0.0 0.5 26864 12192 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7772 0.0 0.5 26888 12276 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7773 0.0 0.5 26892 12196 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7774 1.2 0.5 26892 12240 ? R 10:20 0:02 /usr/sbin/httpd
    apache 7775 0.0 0.5 26864 12188 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7777 0.9 0.5 26896 12260 ? R 10:20 0:01 /usr/sbin/httpd
    apache 7779 0.1 0.5 26876 12188 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7781 0.1 0.5 26864 12208 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7783 20.4 0.5 26832 12208 ? R 10:20 0:39 /usr/sbin/httpd
    apache 7784 0.1 0.5 26892 12244 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7787 0.1 0.5 26888 12236 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7792 0.0 0.5 26872 12220 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7794 0.0 0.5 27000 12312 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7795 0.0 0.5 26880 12228 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7797 0.0 0.5 26892 12232 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7798 0.0 0.5 26888 12236 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7799 0.1 0.5 26888 12220 ? S 10:20 0:00 /usr/sbin/httpd
    apache 7800 0.0 0.5 26864 12184 ? S 10:20 0:00 /usr/sbin/httpd







    I'm not exactly sure what's going on but yesterday, my site was running totally fine. No 20 httpd processes running at once, and my site was running smoothly.

    But just as I dreaded, the site is back to where it was before yesterday. I'm at work at the moment, and I constantly need to restart apache to get my server running again. I've looking at the processes running via top, and I noticed some httpd processes are chewing up a hell of alot of CPU:

    check it out:

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    4386 apache 25 0 27496 11m 7916 R 83.0 0.6 0:52.79 httpd
    4362 apache 25 0 27536 12m 7920 R 49.8 0.6 0:20.66 httpd
    4365 apache 25 0 27496 11m 7900 R 49.8 0.6 0:29.91 httpd
    2193 mysql 15 0 1779m 538m 3084 S 11.3 26.7 1:07.20 mysqld
    4323 apache 15 0 27476 11m 7860 S 1.0 0.6 0:00.35 httpd
    4318 apache 15 0 27480 11m 7868 S 0.7 0.6 0:00.22 httpd
    4312 apache 15 0 27476 11m 7860 S 0.3 0.6 0:00.28 httpd
    4316 apache 15 0 27488 11m 7868 S 0.3 0.6 0:00.24 httpd
    4319 apache 15 0 27476 11m 7864 S 0.3 0.6 0:00.29 httpd
    4355 apache 15 0 27500 11m 7872 S 0.3 0.6 0:00.27 httpd
    4361 apache 15 0 27464 11m 7868 S 0.3 0.6 0:00.33 httpd
    4366 apache 15 0 27480 11m 7872 S 0.3 0.6 0:00.29 httpd
    4367 apache 15 0 27476 11m 7868 S 0.3 0.6 0:00.28 httpd
    4368 apache 15 0 27476 11m 7860 S 0.3 0.6 0:00.32 httpd
    4373 apache 15 0 27476 11m 7868 S 0.3 0.6 0:00.31 httpd
    4375 apache 15 0 27480 11m 7864 S 0.3 0.6 0:00.22 httpd
    4376 apache 16 0 27488 11m 7860 S 0.3 0.6 0:00.27 httpd
    4378 apache 16 0 27476 11m 7864 S 0.3 0.6 0:00.24 httpd
    4385 apache 15 0 27476 11m 7860 S 0.3 0.6 0:00.29 httpd
    4795 apache 16 0 27508 11m 7868 S 0.3 0.6 0:00.16 httpd
    4817 root 16 0 2188 924 724 R 0.3 0.0 0:00.28 top
    4889 apache 16 0 27472 11m 7856 S 0.3 0.6 0:00.14 httpd
    4895 apache 15 0 27436 11m 7860 S 0.3 0.6 0:00.12 httpd



    I've checked my error_logs, and no errors are occurring out of the ordinary. Does any one have a clue about whats going on?

    IonCube, I appreciate your replies. Yes, all the customers do access their stats via the same server, that records the stats. The point I can't get passed is that, why it's happening in the morning and not in the night time? The night time experiences the same amount of traffic as the morning, why would it occur in the morning and not during the night? The server and the mysql config is set up exactly the same at both times, if it's running smoothly at night when the server experiences the same amoutn of traffic, why is dying out in the morning? Would anyone suspect foul play? If so, how can I tell?

    One other thing is that, the page that contains the stats for my users is the main one that's causing the hold up i believe. If you navigate through my site, all the other pages seem to load up lightning fast, but when you do try to access the page that contains the user stats, that's when slow down begins. Obviously, this has some connection with the mysql db, but what could it be? I'm pretty sure I have my indices on the correct columns and also, I have a cron job set up to do a daily optimize of the tables each night.
     
    zonequestz, Feb 28, 2006 IP
  8. blinxdk

    blinxdk Peon

    Messages:
    660
    Likes Received:
    27
    Best Answers:
    0
    Trophy Points:
    0
    #8
    One thing that springs to mind is that you have lots of mysql threads created and none cached, try upping the value for thread_cache in my.cnf

    I've read a suggestion of table_cache = opened table / max_used_connection which would make it 29 for you.

    Also this one:

    Created_tmp_disk_tables | 776 |

    I suggest you try and raise tmp_table_size to try and keep the tmp tables in memory. It's not an alarming number compated to the created_tmp_tables but I'd try and up it a bit to see what happens.
     
    blinxdk, Feb 28, 2006 IP
  9. zonequestz

    zonequestz Guest

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9


    blinxdk, I've made the updates you requested.

    set-variable = thread_cache_size = 29
    set-variable = tmp_table_size = 103554432
    set-variable = table_cache = 6090004 (doubled the table_cache)

    Let's how tomorrow turns out, I appreciate your help! thanks.
     
    zonequestz, Feb 28, 2006 IP
  10. kpatelr

    kpatelr Peon

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    zonequestz,

    I am having the same issue right now and I am restarting the Apache over and over.

    did you set up the start server parameter?

    did this issue was resolved for you?

    really appreciate your response.
     
    kpatelr, Oct 2, 2007 IP
  11. soEvo

    soEvo Peon

    Messages:
    252
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #11
    You could always go to a managed server (where they look after it for you), can recommend a good one (as i work for them :p)
     
    soEvo, Oct 4, 2007 IP
  12. Ladadadada

    Ladadadada Peon

    Messages:
    382
    Likes Received:
    36
    Best Answers:
    0
    Trophy Points:
    0
    #12
    I've had a very similar problem to the one you mention and the cause was excessively long MySQL queries.

    Each Apache process opens up a connection to the database server and then waits 30 or 40 seconds to get it's result back. As you can imagine, at that rate, Apache hits MaxClients very quickly and the site appears to go slow for some people and timeout for others. You can gain some extra time by raising MaxClients but you won't gain very much extra time that way.

    The solution for us was to look through the MySQL slow logs for frequent queries that took a long time (mysqldumpslow is a great tool for analysing those...) and optimise the main culprits. (Then we went through the subversion blame logs to find out who wrote all the dodgy queries :p)
     
    Ladadadada, Oct 5, 2007 IP