How to trouble shoot slow down issues?

Discussion in 'Site & Server Administration' started by ~kev~, Feb 3, 2010.

  1. #1
    About 3 weeks ago my dedicated server was upgraded to a quad core Intel with 8 gigs of memory. On that server I'am running 3 vbulletin forums, and about 5 wordpress blogs.

    Out of all of those sites, only one gets any real traffic.

    The main site gets about 6,000 - 8,000 unique ip addresses daily and around 3.7 - 3.8 million pageviews monthly. The traffic is directly measured with google analytics. This is a vbulletin forum with 1.27 million post, 27,900 members, and 91,000 threads. Database size is about 1.2 gigs.

    All of the other sites get around 200 - 400 unique ip address hits daily.

    As long as the main site has around 300 - 500 people on it, everything is fine. Once we get 600+, and especially 700+ people on the site, it starts slowing down to 5+ second response times. Once we reach 700+ people, response times can reach 8 - 10 seconds.

    Here are some system stats:

    Server Type: Linux CentOS
    Web Server: Apache v2.0.63 (cgi)
    PHP: 5.2.11
    PHP Max Post Size: 16.00 MB
    PHP Maximum Upload Size: 100.00 MB
    PHP Memory Limit: 128.00 MB
    MySQL Version: 5.0.89-community
    MySQL Packet Size: 64.00 MB

    How do I go about trouble shooting where the bottle neck is at?

    What can I do to improve performance?
     
    Last edited: Feb 3, 2010
    ~kev~, Feb 3, 2010 IP
  2. RHS-Chris

    RHS-Chris Well-Known Member

    Messages:
    1,007
    Likes Received:
    35
    Best Answers:
    10
    Trophy Points:
    150
    #2
    Do you have any php caching installed at all? When you server is slow, what do your 'top' stats say? Is there a control panel with it as well? Have you done any tweaking to the my.cnf or to apache? Usually what happens, is that something get hung up, and then you run out of ram, and start using the hard drive, which, slows things down considerably.

    I'd be more than happy to look further into this for you, just shoot me a pm.
     
    RHS-Chris, Feb 3, 2010 IP
  3. wgcom

    wgcom Peon

    Messages:
    157
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    check your iowaits, whats your storage setup?
    check mysql for slow queries, may be archive old forum posts...
    btw, instead of having this monster I would go for separate dedicated machine for mysql and one or two front servers with alot lower specs

    also, check whats going on with the swap when you hit load
     
    Last edited: Feb 3, 2010
    wgcom, Feb 3, 2010 IP
  4. mellow-h

    mellow-h Peon

    Messages:
    750
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Are you using the default parameters for apache and mysql? PHP Configuration doesn't have anything to deal with the performance. What matters are the apache and mysql. They should be well optimized to handle the load. 8GB Ram seems acceptable for traffic near around 800, but definitely not with the default configuration.
     
    mellow-h, Feb 3, 2010 IP
  5. hans

    hans Well-Known Member

    Messages:
    2,923
    Likes Received:
    126
    Best Answers:
    1
    Trophy Points:
    173
    #5
    look with your admin tools what process is using up your resources.

    then ask yourself:
    do you really need those limits?

    use such HIGH limits only if you really need such limits = if you really have files of that size

    you may preserve resources by trying lowest possible limits and see if all still running. example

    PHP Max Post Size: 1.00 MB
    larger limits only make sense if you either have images or videos or similar large files - regular posts on forum and blogs usually are in the 100kb range

    PHP Maximum Upload Size: 1.00 MB
    see above

    PHP Memory Limit: 32.00 MB or 16.00MB
    most php scripts will run with far less than your 128MB
     
    hans, Feb 3, 2010 IP
  6. ~kev~

    ~kev~ Well-Known Member

    Messages:
    2,866
    Likes Received:
    194
    Best Answers:
    0
    Trophy Points:
    110
    #6
    Thanks for the reply guys.

    MYSQL - Here is the mysql config file. This is the same config that I used on the previous dedicated server, which had 4 gigs of memory and a dual core intel CPU.


    PHP - The php memory settings are high for 2 reasons:

    1 - 2 of my forums have a downloads section. Some of the files that are uploaded are close to the 100 meg range.

    2 - The main forum runs a site map generator script. With the default php memory settings of around 8 megs, the sitemap script would consume all of the PHP memory. In order to get the sitemap script to run, the amount of memory allocated to PHP had to be increased. Decreasing the amount of PHP memory, would mean that certain scripts may not run properly.
     
    ~kev~, Feb 3, 2010 IP
  7. wgcom

    wgcom Peon

    Messages:
    157
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    you have some cache solutions installed for php?
    what output "msql>\s" has?
    you use innodb or myisam as table engine?
    whats eating most of your memory?
    what apache mod installed? worker or prefork?
    whats your apache config? threads, clients, keepalive etc.
    whats your storage setup?
    whats going on with swap usage?

    how you measure how many people you have on site? like last active in five mins? or you count number of sessions? whats your session timeout then?
    do you store php sessions in db?
     
    wgcom, Feb 3, 2010 IP
  8. RHS-Chris

    RHS-Chris Well-Known Member

    Messages:
    1,007
    Likes Received:
    35
    Best Answers:
    10
    Trophy Points:
    150
    #8
    Here is a my.cnf file that I use on one of the machines I admin. The server also has 8gb of ram.

    [client]
    port            = 3306
    socket          = /var/lib/mysql/mysql.sock
    
    [mysqld]
    port            = 3306
    socket          = /var/lib/mysql/mysql.sock
    skip-locking
    key_buffer = 250M
    max_allowed_packet = 1M
    table_cache = 8000
    sort_buffer_size = 2M
    read_buffer_size = 2M
    read_rnd_buffer_size = 8M
    myisam_sort_buffer_size = 64M
    thread_cache_size = 8
    query_cache_size = 70M
    thread_concurrency = 8
    max_connections=300
    wait_timeout=120
    query_cache_min_res_unit = 1024
    skip-name-resolve
    thread_cache_size = 12
    local-infile=0
    skip-networking
    
    [mysqldump]
    quick
    max_allowed_packet = 16M
    
    [mysql]
    no-auto-rehash
    
    [isamchk]
    key_buffer = 128M
    sort_buffer_size = 128M
    read_buffer = 2M
    write_buffer = 2M
    
    [myisamchk]
    key_buffer = 128M
    sort_buffer_size = 128M
    read_buffer = 2M
    write_buffer = 2M
    
    [mysqlhotcopy]
    interactive-timeout
    
    Code (markup):
    I would recommend as well, that you use a tuning utility, such as mysql tuning-primer and/or mysqlreport.
     
    RHS-Chris, Feb 3, 2010 IP
  9. RectangleMan

    RectangleMan Notable Member

    Messages:
    2,825
    Likes Received:
    132
    Best Answers:
    0
    Trophy Points:
    210
    #9
    You would benefit from innodb at that size. It would help with slow queries and table locks. Innodb on the users table would be a benefit and you could run the sessions of memory if it's not already. I'd need phpmyadmin to review mysql.

    Your problem could also be apache related. I suggest you run xcache too. Upgrade Apache to 2.2 as it forks/threads much better than 2.0.

    Do you have keep alive on or off for Apache? Have you considered moving the files to another server or a filesharing account? I guess this depends on how often these downloads are occuring.

    Realistically you probably need a forum/server pro to login to your server and examine things while you're having problems. Basic forensics. You could have just one problem of five. Very hard to tell at first. Things needs to be disected.

    If you want to give me SSH Kev let me know. I'll be happy to watch server for a bit and help you optimize the server a little.
     
    RectangleMan, Feb 5, 2010 IP