1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How to find which make a very high load average

Discussion in 'Site & Server Administration' started by upnow, Jan 27, 2010.

  1. #1
    Load average of my server is about 1, but when something happens, it will go up to 100 very quickly, then the system freezes. Happen 1 or 3 time every day. Can happens at any random time, Now I only can write a script to reboot the server when it happen. But I can not find why and how it happen.

    System:Centos5.4 64bit+ Apache+PHP+Mysql

    Anyone has any suggestion.
     
    upnow, Jan 27, 2010 IP
  2. schoash

    schoash Active Member

    Messages:
    291
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    55
    #2
    define "when something happens"
     
    schoash, Jan 27, 2010 IP
  3. upnow

    upnow Peon

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I don't know "something", i just want to know how to find what is the "something", which script make the problem or someone try to crack my server.
     
    Last edited: Jan 27, 2010
    upnow, Jan 27, 2010 IP
  4. RHS-Chris

    RHS-Chris Well-Known Member

    Messages:
    1,007
    Likes Received:
    35
    Best Answers:
    10
    Trophy Points:
    150
    #4
    Is this a cPanel server, or what other control panel, if any, do you use?
     
    RHS-Chris, Jan 27, 2010 IP
  5. taminder

    taminder Peon

    Messages:
    581
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #5
    did you maybe uninstall apache before?

    that happened to me before when I uninstalled apache incorrectly.
     
    taminder, Jan 27, 2010 IP
  6. hostechsupport

    hostechsupport Well-Known Member

    Messages:
    413
    Likes Received:
    23
    Best Answers:
    7
    Trophy Points:
    138
    #6
    Hello,

    I would suggest you write a shell script which will send you top and ps -aufx o/p when load goes beyond certain limit. This would help you to track the load consuming application orelse installing monitoring tool like nagios would help you.

    Thanks
     
    hostechsupport, Jan 27, 2010 IP
  7. upnow

    upnow Peon

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    CentOS 5.3, 8GRAM, Plesk 9.3 installed when ordered the server, no reinstall.

    I have wrote a script to record the top load average every 1 minute. But don't know how to find more info.

    Today we have to decrease the parameter in httpd.conf as below,

    MaxKeepAliveRequests 60
    KeepAliveTimeout 3
    StartServers 25
    MinSpareServers 10
    MaxSpareServers 25
    ServerLimit 450
    MaxClients 400
    MaxRequestsPerChild 2000

    Good new is This problem has still not happened for 1 day, I am still checking if it will happen again,
    Bad new is the traffic is down.
     
    upnow, Jan 27, 2010 IP
  8. upnow

    upnow Peon

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #8
    happen again after 35 hours.:confused:
     
    upnow, Jan 27, 2010 IP
  9. w810i

    w810i Banned

    Messages:
    229
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    53
    #9
    You can try switching to higher server.

    Have you consider it?
     
    w810i, Jan 28, 2010 IP
  10. upnow

    upnow Peon

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    upgrade one time this month, but problem still exists.
     
    upnow, Jan 28, 2010 IP
  11. sumeetrajaggarwal

    sumeetrajaggarwal Peon

    Messages:
    185
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #11
    Ask your provider to investigate about this issue maybe they can help you.
     
    sumeetrajaggarwal, Jan 28, 2010 IP
  12. RHS-Chris

    RHS-Chris Well-Known Member

    Messages:
    1,007
    Likes Received:
    35
    Best Answers:
    10
    Trophy Points:
    150
    #12
    You should start looking in your log files for anything out of the ordinary. I would start with your apache error log file, and see if something is happening around the times when the server binds up.
     
    RHS-Chris, Jan 28, 2010 IP
  13. upnow

    upnow Peon

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #13
    top
    top - 20:03:29 up 6 min, 2 users, load average: 286.48, 159.99, 66.59
    Tasks: 425 total, 298 running, 127 sleeping, 0 stopped, 0 zombie
    Cpu(s): 99.8%us, 0.2%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
    Mem: 8165972k total, 1037708k used, 7128264k free, 80512k buffers
    Swap: 8385920k total, 0k used, 8385920k free, 276280k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    3393 apache 25 0 320m 14m 3296 R 3.3 0.2 0:05.71 httpd
    3397 apache 25 0 320m 15m 4188 R 3.3 0.2 0:05.79 httpd
    3399 apache 25 0 320m 14m 3272 R 3.3 0.2 0:05.71 httpd
    3400 apache 25 0 320m 14m 3200 R 3.3 0.2 0:05.80 httpd
    3402 apache 25 0 320m 14m 3272 R 3.3 0.2 0:05.59 httpd
    3403 apache 25 0 319m 15m 4084 R 3.3 0.2 0:05.59 httpd
    3404 apache 25 0 320m 14m 3272 R 3.3 0.2 0:06.04 httpd
    3405 apache 25 0 320m 15m 4296 R 3.3 0.2 0:05.81 httpd
    3408 apache 25 0 320m 14m 3264 R 3.3 0.2 0:05.69 httpd
    3411 apache 25 0 320m 14m 3260 R 3.3 0.2 0:05.71 httpd
    3415 apache 25 0 321m 15m 4264 R 3.3 0.2 0:05.59 httpd
    3416 apache 25 0 320m 14m 3252 R 3.3 0.2 0:05.74 httpd
    3418 apache 25 0 320m 14m 3144 R 3.3 0.2 0:05.60 httpd
    3420 apache 25 0 320m 15m 3980 R 3.3 0.2 0:05.60 httpd
    3422 apache 25 0 320m 14m 3160 R 3.3 0.2 0:05.89 httpd
    3424 apache 25 0 320m 14m 3136 R 3.3 0.2 0:05.72 httpd
    3510 apache 25 0 320m 14m 3104 R 3.3 0.2 0:05.99 httpd
    3543 apache 25 0 320m 14m 3100 R 3.3 0.2 0:05.79 httpd
    3546 apache 25 0 319m 12m 1948 R 3.3 0.2 0:05.74 httpd
    3610 apache 25 0 322m 17m 5228 R 3.3 0.2 0:06.09 httpd
    3612 apache 25 0 320m 14m 3100 R 3.3 0.2 0:05.79 httpd
    3613 apache 25 0 322m 16m 4188 R 3.3 0.2 0:05.84 httpd
    3615 apache 25 0 322m 16m 4568 R 3.3 0.2 0:06.11 httpd
    3617 apache 25 0 320m 14m 3120 R 3.3 0.2 0:06.10 httpd
    3649 apache 25 0 320m 14m 3120 R 3.3 0.2 0:06.01 httpd
    3653 apache 25 0 320m 14m 3132 R 3.3 0.2 0:05.69 httpd
    3654 apache 25 0 322m 16m 5148 R 3.3 0.2 0:05.89 httpd
    3659 apache 25 0 320m 14m 3116 R 3.3 0.2 0:06.00 httpd
    3664 apache 25 0 319m 14m 3280 R 3.3 0.2 0:05.59 httpd
    3693 apache 25 0 319m 12m 1972 R 3.3 0.2 0:05.90 httpd
    3694 apache 25 0 319m 12m 1940 R 3.3 0.2 0:05.60 httpd
    3699 apache 25 0 323m 16m 3840 R 3.3 0.2 0:06.01 httpd
    3702 apache 25 0 319m 12m 1944 R 3.3 0.2 0:05.59 httpd
    3706 apache 25 0 319m 14m 3336 R 3.3 0.2 0:05.60 httpd
    3716 apache 25 0 320m 14m 3128 R 3.3 0.2 0:06.09 httpd
    3717 apache 25 0 321m 15m 4568 R 3.3 0.2 0:05.73 httpd
    3718 apache 25 0 323m 16m 3788 R 3.3 0.2 0:06.31 httpd
    3721 apache 25 0 321m 14m 3736 R 3.3 0.2 0:05.60 httpd
    3729 apache 25 0 320m 14m 3252 R 3.3 0.2 0:06.10 httpd
    3736 apache 25 0 319m 14m 3264 R 3.3 0.2 0:05.80 httpd
    3746 apache 25 0 319m 14m 3276 R 3.3 0.2 0:05.60 httpd
    3750 apache 25 0 319m 12m 1964 R 3.3 0.2 0:04.39 httpd
    3753 apache 25 0 319m 14m 3280 R 3.3 0.2 0:05.61 httpd
    3755 apache 25 0 319m 12m 1868 R 3.3 0.2 0:04.01 httpd
    3758 apache 25 0 321m 15m 4132 R 3.3 0.2 0:05.59 httpd
    3760 apache 25 0 323m 16m 3792 R 3.3 0.2 0:05.79 httpd
    3761 apache 25 0 321m 14m 3776 R 3.3 0.2 0:05.54 httpd
    3775 apache 25 0 319m 14m 3256 R 3.3 0.2 0:05.99 httpd
    3776 apache 25 0 319m 12m 1932 R 3.3 0.2 0:03.84 httpd
     
    upnow, Jan 28, 2010 IP
  14. chandan123

    chandan123 Prominent Member

    Messages:
    11,586
    Likes Received:
    578
    Best Answers:
    0
    Trophy Points:
    360
    #14
    u running proxies ? what type of sites u have on that server
     
    chandan123, Jan 28, 2010 IP
  15. upnow

    upnow Peon

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #15
    ps -aufx
    Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
    USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
    root 1 0.2 0.0 10348 708 ? Ss 19:57 0:00 init [3]
    root 2 0.0 0.0 0 0 ? S< 19:57 0:00 [migration/0]
    root 3 0.0 0.0 0 0 ? SN 19:57 0:00 [ksoftirqd/0]
    root 4 0.0 0.0 0 0 ? S< 19:57 0:00 [watchdog/0]
    root 5 0.0 0.0 0 0 ? S< 19:57 0:00 [migration/1]
    root 6 0.0 0.0 0 0 ? SN 19:57 0:00 [ksoftirqd/1]
    root 7 0.0 0.0 0 0 ? S< 19:57 0:00 [watchdog/1]
    root 8 0.0 0.0 0 0 ? S< 19:57 0:00 [migration/2]
    root 9 0.0 0.0 0 0 ? SN 19:57 0:00 [ksoftirqd/2]
    root 10 0.0 0.0 0 0 ? S< 19:57 0:00 [watchdog/2]
    root 11 0.0 0.0 0 0 ? S< 19:57 0:00 [migration/3]
    root 12 0.0 0.0 0 0 ? SN 19:57 0:00 [ksoftirqd/3]
    root 13 0.0 0.0 0 0 ? S< 19:57 0:00 [watchdog/3]
    root 14 0.0 0.0 0 0 ? S< 19:57 0:00 [events/0]
    root 15 0.0 0.0 0 0 ? S< 19:57 0:00 [events/1]
    root 16 0.0 0.0 0 0 ? S< 19:57 0:00 [events/2]
    root 17 0.0 0.0 0 0 ? S< 19:57 0:00 [events/3]
    root 18 0.0 0.0 0 0 ? S< 19:57 0:00 [khelper]
    root 91 0.0 0.0 0 0 ? S< 19:57 0:00 [kthread]
    root 98 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [kblockd/0]
    root 99 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [kblockd/1]
    root 100 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [kblockd/2]
    root 101 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [kblockd/3]
    root 102 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [kacpid]
    root 223 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [cqueue/0]
    root 224 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [cqueue/1]
    root 225 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [cqueue/2]
    root 226 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [cqueue/3]
    root 229 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [khubd]
    root 231 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [kseriod]
    root 316 0.0 0.0 0 0 ? S 19:57 0:00 \_ [pdflush]
    root 317 0.0 0.0 0 0 ? S 19:57 0:00 \_ [pdflush]
    root 318 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [kswapd0]
    root 319 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [aio/0]
    root 320 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [aio/1]
    root 321 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [aio/2]
    root 322 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [aio/3]
    root 467 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [kpsmoused]
    root 514 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [ata/0]
    root 515 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [ata/1]
    root 516 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [ata/2]
    root 517 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [ata/3]
    root 518 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [ata_aux]
    root 524 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [scsi_eh_0]
    root 525 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [scsi_eh_1]
    root 526 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [scsi_eh_2]
    root 527 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [scsi_eh_3]
    root 528 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [scsi_eh_4]
    root 529 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [scsi_eh_5]
    root 540 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [kstriped]
    root 561 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [kjournald]
    root 588 0.0 0.0 0 0 ? S< 19:57 0:00 \_ [kauditd]
    root 1851 0.0 0.0 0 0 ? S< 19:58 0:00 \_ [kmpathd/0]
    root 1852 0.0 0.0 0 0 ? S< 19:58 0:00 \_ [kmpathd/1]
    root 1853 0.0 0.0 0 0 ? S< 19:58 0:00 \_ [kmpathd/2]
    root 1854 0.0 0.0 0 0 ? S< 19:58 0:00 \_ [kmpathd/3]
    root 1855 0.0 0.0 0 0 ? S< 19:58 0:00 \_ [kmpath_handlerd]
    root 1884 0.0 0.0 0 0 ? S< 19:58 0:00 \_ [kjournald]
    root 1886 0.0 0.0 0 0 ? S< 19:58 0:00 \_ [kjournald]
    root 2804 0.0 0.0 0 0 ? S< 19:58 0:00 \_ [rpciod/0]
    root 2805 0.0 0.0 0 0 ? S< 19:58 0:00 \_ [rpciod/1]
    root 2807 0.0 0.0 0 0 ? S< 19:58 0:00 \_ [rpciod/2]
    root 2808 0.0 0.0 0 0 ? S< 19:58 0:00 \_ [rpciod/3]
    root 622 0.0 0.0 12672 816 ? S<s 19:57 0:00 /sbin/udevd -d
    root 1972 0.0 0.0 66068 1476 ? Ss 19:58 0:00 /bin/bash /etc/rc.d/rc 3
    root 4355 0.0 0.0 65932 1212 ? S 20:04 0:00 \_ /bin/bash /etc/rc3.d/S95atd start
    root 4356 0.0 0.0 65932 552 ? S 20:04 0:00 \_ /bin/bash /etc/rc3.d/S95atd start
    root 4357 0.0 0.0 65932 300 ? R 20:04 0:00 \_ /bin/bash /etc/rc3.d/S95atd start
    root 2656 0.0 0.0 27332 836 ? S<sl 19:58 0:00 auditd
    root 2658 0.0 0.0 16272 768 ? S<sl 19:58 0:00 \_ /sbin/audispd
    root 2691 0.0 0.0 5908 616 ? Ss 19:58 0:00 syslogd -m 0
    root 2694 0.0 0.0 3804 440 ? Ss 19:58 0:00 klogd -x
    root 2710 0.0 0.0 10760 380 ? Ss 19:58 0:00 irqbalance
    rpc 2732 0.0 0.0 8052 592 ? Ss 19:58 0:00 portmap
    root 2761 0.0 0.0 10144 820 ? Ss 19:58 0:00 rpc.statd
    root 2815 0.0 0.0 48716 728 ? Ss 19:58 0:00 rpc.idmapd
    dbus 2890 0.0 0.0 21260 916 ? Ss 19:58 0:00 dbus-daemon --system
    root 2905 0.0 0.0 10432 800 ? Ss 19:58 0:00 /usr/sbin/hcid
    root 2911 0.0 0.0 5936 548 ? Ss 19:58 0:00 /usr/sbin/sdpd
    root 2940 0.0 0.0 0 0 ? S< 19:58 0:00 [krfcommd]
    root 2990 0.0 0.0 21176 1424 ? Ssl 19:58 0:00 pcscd
    root 3012 0.0 0.0 8516 484 ? Ss 19:58 0:00 /usr/bin/hidd --server
    root 3035 0.0 0.0 32008 1256 ? Ssl 19:58 0:00 automount
    root 3059 0.0 0.0 3800 572 ? Ss 19:58 0:00 /usr/sbin/acpid
    500 3083 0.0 0.0 59224 2084 ? S 19:58 0:00 /usr/sbin/sw-cp-serverd -f /etc/sw-cp-server/config
    root 3100 0.0 0.0 60676 1200 ? Ss 19:58 0:00 /usr/sbin/sshd
    root 3809 0.0 0.0 99516 3728 ? Ss 19:59 0:00 \_ sshd: root@pts/0
    root 3848 0.0 0.0 66064 1572 pts/0 Ss 20:00 0:00 | \_ -bash
    root 4363 0.0 0.0 65924 1348 pts/0 R+ 20:04 0:00 | \_ ps -aufx
    root 3889 0.0 0.0 98732 3744 ? Ss 20:00 0:00 \_ sshd: root@notty
    root 4013 0.0 0.0 54132 2124 ? Ss 20:01 0:00 | \_ /usr/libexec/openssh/sftp-server
    root 3918 0.0 0.0 98732 3728 ? Ss 20:00 0:00 \_ sshd: root@pts/1
    root 3937 0.0 0.0 66064 1584 pts/1 Ss+ 20:00 0:00 \_ -bash
    root 3115 0.0 0.0 135408 2616 ? Ss 19:58 0:00 cupsd
    root 3131 0.0 0.0 21644 908 ? Ss 19:58 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
    root 3151 0.0 0.0 40924 804 ? S 19:58 0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr/sbin/courierlogger -stderrloggername=imapd -maxpr
    root 3153 0.0 0.0 33612 848 ? S 19:58 0:00 /usr/sbin/courierlogger imapd
    root 3161 0.0 0.0 40924 800 ? S 19:58 0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr/sbin/courierlogger -stderrloggername=imapd-ssl -m
    root 3163 0.0 0.0 33612 848 ? S 19:58 0:00 /usr/sbin/courierlogger imapd-ssl
    root 3169 0.0 0.0 40924 800 ? S 19:58 0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr/sbin/courierlogger -stderrloggername=pop3d -maxpr
    root 3171 0.0 0.0 33612 852 ? S 19:58 0:00 /usr/sbin/courierlogger pop3d
    root 3178 0.0 0.0 40924 800 ? S 19:58 0:00 /usr/lib/courier-imap/couriertcpd -address=0 -stderrlogger=/usr/sbin/courierlogger -stderrloggername=pop3d-ssl -m
    root 3180 0.0 0.0 33612 852 ? S 19:58 0:00 /usr/sbin/courierlogger pop3d-ssl
    qmails 3197 0.0 0.0 3848 432 ? S 19:58 0:00 qmail-send
    qmaill 3199 0.0 0.0 3796 524 ? S 19:58 0:00 \_ splogger qmail
    root 3200 0.0 0.0 3836 404 ? S 19:58 0:00 \_ qmail-lspawn | /usr/bin/deliverquota ./Maildir
    qmailr 3201 0.0 0.0 3836 400 ? S 19:58 0:00 \_ qmail-rspawn
    qmailq 3202 0.0 0.0 3792 380 ? S 19:58 0:00 \_ qmail-clean
    root 3221 0.0 0.0 6452 376 ? Ss 19:58 0:00 gpm -m /dev/input/mice -t exps2
    apache 3251 0.0 0.1 232732 9120 ? S 19:58 0:00 /usr/sbin/httpd
    named 3298 0.0 0.0 251868 5876 ? Ssl 19:58 0:00 /usr/sbin/named -u named -c /etc/named.conf -u named -t /var/named/run-root
    apache 3389 1.8 0.1 327904 15084 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3390 1.8 0.1 327904 15096 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3391 1.8 0.1 327904 15860 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3392 1.7 0.1 326880 14580 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3393 1.8 0.1 327904 15080 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3394 1.8 0.1 327904 16120 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3395 1.8 0.1 327904 15068 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3396 1.4 0.1 326904 15768 ? R 19:58 0:04 /usr/sbin/httpd
    apache 3397 1.8 0.1 327904 16040 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3398 1.8 0.1 327904 15076 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3399 1.8 0.1 327904 15056 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3400 1.8 0.1 327904 14988 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3401 1.8 0.1 326880 13060 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3402 1.7 0.1 327904 15056 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3403 1.7 0.1 326880 15696 ? R 19:58 0:05 /usr/sbin/httpd
    apache 3404 1.9 0.1 327904 15056 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3405 1.8 0.1 327904 16136 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3406 1.8 0.1 327904 15068 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3407 1.5 0.1 327904 14972 ? R 19:58 0:05 /usr/sbin/httpd
    apache 3408 1.8 0.1 327904 15048 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3409 1.5 0.1 327904 15872 ? R 19:58 0:05 /usr/sbin/httpd
    apache 3410 1.8 0.1 327928 15592 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3411 1.8 0.1 327904 15044 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3412 1.8 0.1 327904 15080 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3413 1.5 0.2 329996 17108 ? R 19:58 0:05 /usr/sbin/httpd
    apache 3414 1.9 0.2 331012 17100 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3415 1.7 0.1 328972 15716 ? R 19:58 0:05 /usr/sbin/httpd
    apache 3416 1.8 0.1 327904 15032 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3417 1.8 0.1 328964 15656 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3418 1.7 0.1 327904 14928 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3419 1.7 0.1 327904 15644 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3420 1.7 0.1 327904 15772 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3421 1.8 0.1 327904 14932 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3422 1.8 0.1 327904 14944 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3423 1.4 0.1 326904 14896 ? R 19:58 0:04 /usr/sbin/httpd
    apache 3424 1.8 0.1 327904 14920 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3425 1.5 0.2 331020 19156 ? R 19:58 0:05 /usr/sbin/httpd
    apache 3426 1.5 0.2 329980 16728 ? R 19:58 0:05 /usr/sbin/httpd
    apache 3427 1.8 0.1 327904 14932 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3510 1.9 0.1 327904 14884 ? R 19:58 0:06 /usr/sbin/httpd
    postgres 3541 0.0 0.0 120684 3376 ? S 19:58 0:00 /usr/bin/postmaster -p 5432 -D /var/lib/pgsql/data
    postgres 3550 0.0 0.0 109864 700 ? S 19:58 0:00 \_ postgres: logger process
    postgres 3552 0.0 0.0 120684 1004 ? S 19:58 0:00 \_ postgres: writer process
    postgres 3553 0.0 0.0 110864 696 ? S 19:58 0:00 \_ postgres: stats buffer process
    postgres 3554 0.0 0.0 110052 868 ? S 19:58 0:00 \_ postgres: stats collector process
    apache 3543 1.8 0.1 327904 14952 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3544 1.9 0.2 329976 16336 ? R 19:58 0:06 /usr/sbin/httpd
    apache 3545 1.5 0.1 327904 14900 ? R 19:58 0:05 /usr/sbin/httpd
    apache 3546 1.8 0.1 326880 13052 ? R 19:58 0:06 /usr/sbin/httpd
    .... hundreds of httpd like above ....
    apache 4090 1.3 0.1 326880 12764 ? R 20:01 0:01 /usr/sbin/httpd
    apache 4091 1.2 0.1 326880 12756 ? R 20:01 0:01 /usr/sbin/httpd
    apache 4093 1.2 0.1 326880 12748 ? R 20:01 0:01 /usr/sbin/httpd
    root 4103 0.0 0.0 65932 1280 pts/0 S 20:01 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --log-error=/var/log/mys
    mysql 4161 0.0 0.8 477416 70672 pts/0 Sl 20:02 0:00 \_ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pi
    apache 4109 1.2 0.1 326880 12784 ? R 20:01 0:01 /usr/sbin/httpd
    apache 4110 1.8 0.1 326880 13060 ? R 20:01 0:02 /usr/sbin/httpd
    apache 4111 1.1 0.1 326880 12996 ? R 20:01 0:01 /usr/sbin/httpd
    drweb 4276 0.0 0.5 52404 47796 ? Ss 20:02 0:00 drwebd.real
    drweb 4277 0.0 0.5 52404 47644 ? S 20:02 0:00 \_ drwebd.real
    drweb 4278 0.0 0.5 52404 47648 ? S 20:02 0:00 \_ drwebd.real
    drweb 4279 0.0 0.5 52404 47648 ? S 20:02 0:00 \_ drwebd.real
    drweb 4291 0.0 0.5 52404 47648 ? S 20:02 0:00 \_ drwebd.real
    root 4294 0.0 0.0 74816 1172 ? Ss 20:02 0:00 crond
    root 4345 0.0 0.0 133492 2760 ? S 20:04 0:00 \_ crond
    root 4346 0.0 0.0 8700 952 ? Ss 20:04 0:00 \_ /bin/sh -c /root/checkload.sh >> /root/checkload.log
    root 4347 0.0 0.0 8704 980 ? S 20:04 0:00 \_ /bin/sh /root/checkload.sh
    root 4360 0.0 0.0 8704 492 ? R 20:04 0:00 \_ /bin/sh /root/checkload.sh
    root 4361 0.0 0.0 0 0 ? Z 20:04 0:00 \_ [checkload.sh] <defunct>
    root 4362 0.0 0.0 8704 164 ? R 20:04 0:00 \_ /bin/sh /root/checkload.sh
    xfs 4329 0.0 0.0 20084 1040 ? Ss 20:03 0:00 xfs -droppriv -daemon
    root 4353 0.0 0.0 3804 520 ? RNs 20:04 0:00 anacron -s
     
    upnow, Jan 28, 2010 IP
  16. upnow

    upnow Peon

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #16
    One big Game Site and some other type site.
     
    upnow, Jan 28, 2010 IP
  17. upnow

    upnow Peon

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #17
    Happen problem at 19:55:01 ~ 19:56:32

    From message

    Jan 28 19:45:03 kernel: ipt_connlimit: Oops: invalid ct state ?
    Jan 28 19:45:21 last message repeated 30 times
    Jan 28 19:45:21 xinetd[3103]: EXIT: smtp status=1 pid=13702 duration=19(sec)
    Jan 28 19:45:22 kernel: ipt_connlimit: Oops: invalid ct state ?
    Jan 28 19:45:53 last message repeated 74 times
    Jan 28 19:46:54 last message repeated 98 times
    Jan 28 19:47:58 last message repeated 52 times
    Jan 28 19:48:59 last message repeated 77 times
    Jan 28 19:50:00 last message repeated 128 times
    Jan 28 19:51:01 last message repeated 134 times
    Jan 28 19:52:02 last message repeated 141 times
    Jan 28 19:53:04 last message repeated 94 times
    Jan 28 19:54:05 last message repeated 120 times
    Jan 28 19:55:06 last message repeated 82 times
    Jan 28 19:56:08 last message repeated 105 times
    Jan 28 19:56:57 last message repeated 436 times
    Jan 28 19:56:57 shutdown[14261]: shutting down for system reboot


    The Load Average
    10-01-28 19:45:01 up 14:34, 1 user, load average: 0.10, 0.18, 0.24
    10-01-28 19:46:01 up 14:35, 1 user, load average: 0.32, 0.25, 0.26
    10-01-28 19:47:01 up 14:36, 1 user, load average: 0.36, 0.26, 0.27
    10-01-28 19:48:01 up 14:37, 1 user, load average: 0.28, 0.26, 0.27
    10-01-28 19:49:01 up 14:38, 1 user, load average: 0.15, 0.22, 0.25
    10-01-28 19:50:01 up 14:39, 1 user, load average: 0.05, 0.18, 0.24
    10-01-28 19:51:01 up 14:40, 1 user, load average: 0.19, 0.19, 0.24
    10-01-28 19:52:01 up 14:41, 1 user, load average: 0.17, 0.18, 0.23
    10-01-28 19:53:01 up 14:42, 1 user, load average: 0.18, 0.19, 0.23
    10-01-28 19:54:01 up 14:43, 1 user, load average: 0.21, 0.19, 0.22
    10-01-28 19:55:01 up 14:44, 1 user, load average: 0.15, 0.18, 0.21
    10-01-28 19:56:32 up 14:45, 1 user, load average: 101.60, 31.09, 10.96
    ## 10-01-28 19:56:39 reboot 161
    10-01-28 19:57:01 up 14:46, 2 users, load average: 114.65, 41.40, 15.06
    ## 10-01-28 19:57:01 reboot 7
    10-01-28 20:04:03 up 6 min, 2 users, load average: 290.82, 175.06, 75.09
    ## 10-01-28 20:04:09 reboot 295
    10-01-28 20:05:16 up 7 min, 2 users, load average: 295.07, 200.53, 91.20
    ## 10-01-28 20:05:22 reboot 295
    10-01-28 20:07:02 up 1 min, 0 users, load average: 1.05, 0.29, 0.09
    10-01-28 20:08:01 up 2 min, 2 users, load average: 0.56, 0.28, 0.10
     
    upnow, Jan 28, 2010 IP
  18. upnow

    upnow Peon

    Messages:
    30
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #18
    Maby the problem is the php safe-mode which is set on for some wrong site.
    Disable these sites, has been Ok for 60 hours now.
     
    upnow, Jan 31, 2010 IP