The server is going offline at least 3 times a day!

My problem is that my server hosted on fdcservers.net is going offline at least 3 times a day. then i have to contact their stuff to reboot and it's ok for an hour or so?

Can anybody help me locate error logs where I can see what's wrong please.

Kind regards.

 

You need to check

/var/log/messages

Also

/var/logs/httpd/error_log

 

It gives me permition denied message undoe I'm loged as root

 

Got the messages

Feb 24 17:05:49 talented sshd(pam_unix)[8968]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:05:52 talented sshd(pam_unix)[8970]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:05:55 talented sshd(pam_unix)[8972]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:05:57 talented sshd(pam_unix)[8974]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:00 talented sshd(pam_unix)[8976]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:03 talented sshd(pam_unix)[8978]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:06 talented sshd(pam_unix)[8980]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:09 talented sshd(pam_unix)[8983]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:12 talented sshd(pam_unix)[8986]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:14 talented sshd(pam_unix)[8992]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:17 talented sshd(pam_unix)[8994]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:20 talented sshd(pam_unix)[8996]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:23 talented sshd(pam_unix)[8998]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:25 talented sshd(pam_unix)[9000]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:28 talented sshd(pam_unix)[9002]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:31 talented sshd(pam_unix)[9004]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:34 talented sshd(pam_unix)[9007]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:37 talented sshd(pam_unix)[9009]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:39 talented sshd(pam_unix)[9014]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:42 talented sshd(pam_unix)[9016]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:45 talented sshd(pam_unix)[9018]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:48 talented sshd(pam_unix)[9020]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:51 talented sshd(pam_unix)[9022]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:53 talented sshd(pam_unix)[9024]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:56 talented sshd(pam_unix)[9026]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:59 talented sshd(pam_unix)[9028]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:02 talented sshd(pam_unix)[9030]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:05 talented sshd(pam_unix)[9033]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:08 talented sshd(pam_unix)[9038]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:10 talented sshd(pam_unix)[9044]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:31 talented sshd(pam_unix)[9004]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:34 talented sshd(pam_unix)[9007]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:37 talented sshd(pam_unix)[9009]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:39 talented sshd(pam_unix)[9014]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:42 talented sshd(pam_unix)[9016]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:45 talented sshd(pam_unix)[9018]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:48 talented sshd(pam_unix)[9020]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:51 talented sshd(pam_unix)[9022]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:53 talented sshd(pam_unix)[9024]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:56 talented sshd(pam_unix)[9026]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:06:59 talented sshd(pam_unix)[9028]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:02 talented sshd(pam_unix)[9030]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:05 talented sshd(pam_unix)[9033]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:08 talented sshd(pam_unix)[9038]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:10 talented sshd(pam_unix)[9044]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:13 talented sshd(pam_unix)[9046]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:16 talented sshd(pam_unix)[9049]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:19 talented sshd(pam_unix)[9052]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:22 talented sshd(pam_unix)[9054]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:25 talented sshd(pam_unix)[9056]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:27 talented sshd(pam_unix)[9058]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:30 talented sshd(pam_unix)[9060]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:33 talented sshd(pam_unix)[9062]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:36 talented sshd(pam_unix)[9064]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root
Feb 24 17:07:39 talented sshd(pam_unix)[9066]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=plesk.okchost2.net user=root


I have lot's of this messages. what is this?

 

It seems a bruteforce attack :/

 

hmm... i think the server doesn't go offline because of this. It went down again just now. I sended a msg to reboot the machine to fdcservers stuff. I will take a look what was wrong in the log

 

If the bruteforce attack is huge it could take down the server, check the logs when the server goes down.

 

Also do a top command (just put top trough ssh) to check the processes running in your server.

 

OK got it! This is the log from befire it got restarted. Don't see anything usefull here

Feb 29 10:32:19 talented kernel: kjournald starting. Commit interval 5 seconds
Feb 29 10:32:19 talented kernel: EXT3-fs warning: maximal mount count reached, running e2fsck is recommended
Feb 29 10:32:19 talented kernel: EXT3 FS 2.4-0.9.19, 19 August 2002 on loop(7,0), internal journal
Feb 29 10:32:19 talented kernel: EXT3-fs: recovery complete.
Feb 29 10:32:19 talented kernel: EXT3-fs: mounted filesystem with ordered data mode.
Feb 29 10:38:57 talented proftpd[11690]: talented.***.com (89.212.34.226[89.212.34.226]) - FTP session opened.
Feb 29 10:39:00 talented proftpd[11690]: talented.***.com (89.212.34.226[89.212.34.226]) - FTP session closed.
Feb 29 11:02:10 talented named[10111]: client 202.157.182.142#4701: transfer of 'top-myspace.com/IN': AXFR started
Feb 29 11:45:09 talented sshd(pam_unix)[20477]: session opened for user root by (uid=0)
Feb 29 12:09:38 talented syslogd 1.4.1: restart.
Feb 29 12:09:38 talented syslog: syslogd startup succeeded
Feb 29 12:09:38 talented kernel: klogd 1.4.1, log source = /proc/kmsg started.
Feb 29 12:09:38 talented syslog: klogd startup succeeded
Feb 29 12:09:38 talented kernel: Linux version 2.4.20-8 (bhcompile@stripples.devel.redhat.com) (gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)) #1 Thu Mar 13 17:18:24 EST 2003
Feb 29 12:09:38 talented kernel: BIOS-provided physical RAM map:
Feb 29 12:09:38 talented kernel: BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
Feb 29 12:09:38 talented kernel: BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
Feb 29 12:09:38 talented kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Feb 29 12:09:38 talented kernel: BIOS-e820: 0000000000100000 - 000000005dff0000 (usable)
Feb 29 12:09:38 talented kernel: BIOS-e820: 000000005dff0000 - 000000005dff8000 (ACPI data)
Feb 29 12:09:38 talented kernel: BIOS-e820: 000000005dff8000 - 000000005e000000 (ACPI NVS)
Feb 29 12:09:38 talented kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
Feb 29 12:09:38 talented kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Feb 29 12:09:38 talented kernel: BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
Feb 29 12:09:38 talented keytable: Loading keymap:
Feb 29 12:09:38 talented kernel: 607MB HIGHMEM available.
Feb 29 12:09:38 talented keytable:
Feb 29 12:09:38 talented kernel: 896MB LOWMEM available.
Feb 29 12:09:38 talented keytable: Loading system font:

 

I don't see anything, but i don't know what does this means:607MB HIGHMEM available & 896MB LOWMEM available, maybe you are out of memory? Do a free -m to check used memory, anyways make sure your system is updated (use yum update to check).

 

I have thousand of SSH bruteforce attack everyday. It does not crash my machine. I think it is something else. If you find out the bruteforce attack original IP. You can ask iptables to drop packet coming from that IP.

 

total used free shared buffers cached
Mem: 1480 731 748 0 41 503
-/+ buffers/cache: 186 1294
Swap: 956 0 956


Does anybody has another idea? I updated system and server software via WHM.

 

Is there anybody to help me out. I will pay for the work done

 

It is not always easy to find why server whent down by checking the log files. If you are running top while it goes down, you will be able to find which process take more cpu and crash the server. If you install csf firewall, it have lfd, that will ban IP that do brute force, also mail you when server load is high, this can caused to too many hits (access) to some of your web site.

 

make the websites ip dedicated or not shared with the main ip of the server.

 

I can work on it tomorrow, in case it still is a problem.

 

FDC solved the problem. The RAM was not ok!

 

In case there is a further brute force attack, you may consider having an APF installed over your machine

 

fdcservers.net say that I don't have enough RAM. But it was enough last 2 years and the traffic and transferr dropped in past few months.

Don't know what I don't need here and I can uninstall?

User Domain %CPU %MEM MySQL Processes
munin 8.84 0.46 0.0
Top Process %CPU 27.0 /usr/bin/perl -w /usr/share/munin/munin-update
Top Process %CPU 23.0 /usr/bin/perl -w /usr/share/munin/munin-update
Top Process %CPU 22.0 /usr/bin/perl -w /usr/share/munin/munin-update
jernejm1 nejcpass.com 2.17 2.89 0.0
Top Process %CPU 32.7 /usr/bin/perl /usr/local/cpanel/3rdparty/bin/awstats.pl -config -update
Top Process %CPU 29.4 /usr/bin/perl /usr/local/cpanel/3rdparty/bin/awstats.pl -config -update
Top Process %CPU 8.5 [spamd]
root 13.13 19.26 2.0
Top Process %CPU 37.0 /usr/bin/perl /usr/bin/mrtg /etc/mrtg/mrtg.cfg
Top Process %CPU 32.0 /usr/bin/perl /usr/bin/mrtg /etc/mrtg/mrtg.cfg
Top Process %CPU 31.0 /usr/bin/perl /usr/bin/mrtg /etc/mrtg/mrtg.cfg
nobody 0.95 37.85 0.0
Top Process %CPU 9.4 [httpd]
Top Process %CPU 9.0 [httpd]
Top Process %CPU 6.1 [httpd]
mailnull 0.59 0.96 0.0
Top Process %CPU 5.0 [exim]
Top Process %CPU 2.6 [exim]
Top Process %CPU 1.6 [exim]

Please help!

 

Can you show the result of top command? (a screen if possible)