Cpanel server overload (slow write sql performance)

Discussion in 'Site & Server Administration' started by ASTRAPI, Dec 11, 2010.

  1. #1
    Hello

    On a dedicated server:

    
    2x Intel Xeon Quad 5405
    16GB Ram
    4x sas raid 10 disks 15.000rpm
    1.000Mbit 
    Code (markup):
    I got only one account ( i do not plan to use another account on the server) inside with a big invision forum (300 users online per minute) and i use cpanel.

    
    Apache 2.2.17 (php handler: fcgi )- (apache suexec: off)
    PHP 5.2.14
    Mysql 5.1.51
    And i use nginx plugin in front of apache
    
    Code (markup):
    My problem is that i have high server load in the afternoons about 14-15 and some stats here:

    [​IMG]

    and from top command:

    [​IMG]

    vmstat command:

    [​IMG]

    uname -r command:

    2.6.18-194.17.1.el5
    Code (markup):
    iostat comand:

    http://i54.tinypic.com/2vaavi8.png

    iostat -d -x 5 3 command:

    
    
    Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
    sda               0.01    61.62  0.09 72.21     3.09  1070.92    14.85     0.07    0.95   0.04   0.27
    sda1              0.00     2.04  0.00  1.03     0.12    24.61    23.84     0.00    0.34   0.13   0.01
    sda2              0.01     8.72  0.02  2.76     1.47    91.82    33.59     0.00    0.41   0.11   0.03
    sda3              0.00    33.49  0.06 65.40     0.89   791.30    12.10     0.06    0.95   0.03   0.23
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     2.00     0.00   35.25  35.25   0.00
    sda5              0.00     0.00  0.00  0.00     0.00     0.00    24.88     0.00    4.15   3.51   0.00
    sda6              0.00     0.60  0.00  0.20     0.00     6.40    32.30     0.00    0.44   0.11   0.00
    sda7              0.00    16.78  0.01  2.82     0.62   156.79    55.55     0.00    1.76   0.07   0.02
    
    Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
    sda               0.00    46.20  0.20 32.00     3.20   625.60    19.53     0.02    0.67   0.12   0.38
    sda1              0.00     1.40  0.20  0.60     3.20    16.00    24.00     0.00    3.00   3.00   0.24
    sda2              0.00    15.80  0.00  2.40     0.00   145.60    60.67     0.00    0.00   0.00   0.00
    sda3              0.00    14.20  0.00 28.20     0.00   339.20    12.03     0.02    0.67   0.04   0.12
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sda5              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sda6              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sda7              0.00    14.80  0.00  0.80     0.00   124.80   156.00     0.00    0.25   0.25   0.02
    
    Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
    sda               0.00   141.00  0.00 196.80     0.00  2704.00    13.74     0.26    1.33   0.02   0.38
    sda1              0.00     5.20  0.00  0.60     0.00    46.40    77.33     0.00    0.00   0.00   0.00
    sda2              0.00    14.20  0.00  2.20     0.00   131.20    59.64     0.00    0.64   0.09   0.02
    sda3              0.00    87.80  0.00 174.20     0.00  2097.60    12.04     0.24    1.39   0.02   0.36
    sda4              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sda5              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
    sda6              0.00     1.40  0.00  0.40     0.00    14.40    36.00     0.00    0.00   0.00   0.00
    sda7              0.00    32.40  0.00 19.40     0.00   414.40    21.36     0.02    0.96   0.01   0.02
    
    Code (markup):
    hdparm -Tt (device) :
    
    
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda1              48G  670M   45G   2% /
    /dev/sda7              87G  1.4G   81G   2% /home
    /dev/sda6             9.5G  152M  8.9G   2% /tmp
    /dev/sda3              48G  2.4G   43G   6% /var
    /dev/sda2              48G  4.5G   41G  10% /usr
    tmpfs                 7.9G     0  7.9G   0% /dev/shm
    
    
    hdparm -Tt /dev/sda2 command:
    
    /dev/sda2:
     Timing cached reads:   13892 MB in  2.00 seconds = 6957.42 MB/sec
     Timing buffered disk reads:  410 MB in  3.01 seconds = 136.26 MB/sec
    root@server [~]# hdparm -Tt /dev/sda3
    
    /dev/sda3:
     Timing cached reads:   13636 MB in  2.00 seconds = 6828.73 MB/sec
     Timing buffered disk reads:  1122 MB in  3.00 seconds = 373.88 MB/sec
    root@server [~]# hdparm -Tt /dev/sda7
    
    /dev/sda7:
     Timing cached reads:   10620 MB in  1.98 seconds = 5365.54 MB/sec
     Timing buffered disk reads:  688 MB in  3.02 seconds = 227.56 MB/sec
    Code (markup):
    mysqladmin processlist

    http://i56.tinypic.com/2e3v2jb.png

    ibf_posts that have the most entries about 1.300.000 seems to be the problem on my write performance to sql and here maybe is my biggest problem...

    I am thinking also to use a second disk for sql only if that will help a lot so mysql will be able to serv much more requests....


    mysqladmin processlist |wc -l command:

    54
    Code (markup):
    netstat -an |grep :80 |wc -l command

    1557 <-- sometimes i have more there.
    Code (markup):

    my.cnf:

    
    [mysqld]
    skip-innodb
    skip-locking
    skip-networking
    thread_concurrency=16
    safe-show-database
    tmp_table_size=256M
    max_heap_table_size=256M
    query_cache_limit=3M
    query_cache_size=128M
    query_cache_type=1
    max_user_connections=300
    max_connections=300
    interactive_timeout=30
    wait_timeout=50
    connect_timeout=30
    low_priority_updates=1
    thread_cache_size=512
    key_buffer=1024M
    join_buffer_size=256M
    max_allowed_packet=64M
    table_cache=1536
    record_buffer=1M
    sort_buffer_size=16M
    read_buffer_size=16M
    max_connect_errors=10
    myisam_sort_buffer_size=512M
    concurrent_insert=2
    
    [isamchk]
    key_buffer=128M
    sort_buffer128M
    read_buffer=32M
    write_buffer=32M
    
    [myisamchk]
    key_buffer=128M
    sort_buffer=128M
    read_buffer=32M
    write_buffer=32M
    
    [mysql]
    no-auto-rehash
    
    Code (markup):
    If you need any other command to run let me know.

    Read performance is amazing les than a second to load anything.
    Write performance about 20-30 seconds and some times i got internal server error at the end.

    I can pay for anything that solve my problem with very slow write sql performance :)

    Thank you
     
    Last edited: Dec 11, 2010
    ASTRAPI, Dec 11, 2010 IP
  2. RHS-Chris

    RHS-Chris Well-Known Member

    Messages:
    1,007
    Likes Received:
    35
    Best Answers:
    10
    Trophy Points:
    150
    #2
    Make sure you have a PHP caching engine on, like eAccelerator. This can be done through EasyApache. In my opinion, that server should handle that site fairly easily. I would also suggest turning KeepAlive off in Apache, as I have found that it helps with lots of connections to the server.

    Chris
     
    RHS-Chris, Dec 11, 2010 IP
  3. ASTRAPI

    ASTRAPI Guest

    Messages:
    500
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    KeepAlive option is not on httpd.conf :(
     
    ASTRAPI, Dec 11, 2010 IP
  4. mubashirnisar

    mubashirnisar Banned

    Messages:
    2,186
    Likes Received:
    22
    Best Answers:
    0
    Trophy Points:
    0
    #4
    in apache 2 and later verswion its not in httpd.conf,you need to find it in its includes,which mpm are you using to handle your apache requests?

    more over you might want to optimize your mysql and install mytop as well
     
    mubashirnisar, Dec 11, 2010 IP
  5. ASTRAPI

    ASTRAPI Guest

    Messages:
    500
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I am using prefork.
     
    ASTRAPI, Dec 11, 2010 IP